* [PATCH v2 01/12] fsmonitor: refactor filesystem checks to common interface
2022-10-14 21:45 ` [PATCH v2 " Eric DeCosta via GitGitGadget
@ 2022-10-14 21:45 ` Eric DeCosta via GitGitGadget
2022-10-14 21:45 ` [PATCH v2 02/12] fsmonitor: relocate socket file if .git directory is remote Eric DeCosta via GitGitGadget
` (13 subsequent siblings)
14 siblings, 0 replies; 89+ messages in thread
From: Eric DeCosta via GitGitGadget @ 2022-10-14 21:45 UTC (permalink / raw)
To: git
Cc: Eric Sunshine, Ævar Arnfjörð Bjarmason,
Eric DeCosta, Eric DeCosta
From: Eric DeCosta <edecosta@mathworks.com>
Provide a common interface for getting basic filesystem information
including filesystem type and whether the filesystem is remote.
Refactor existing code for getting basic filesystem info and detecting
remote file systems to the new interface.
Refactor filesystem checks to leverage new interface. For macOS,
error-out if the Unix Domain socket (UDS) file is on a remote
filesystem.
Signed-off-by: Eric DeCosta <edecosta@mathworks.com>
---
Makefile | 1 +
compat/fsmonitor/fsm-path-utils-darwin.c | 43 ++++++
compat/fsmonitor/fsm-path-utils-win32.c | 128 +++++++++++++++++
compat/fsmonitor/fsm-settings-darwin.c | 69 +++------
compat/fsmonitor/fsm-settings-win32.c | 172 +----------------------
contrib/buildsystems/CMakeLists.txt | 2 +
fsmonitor-path-utils.h | 26 ++++
fsmonitor-settings.c | 50 +++++++
8 files changed, 272 insertions(+), 219 deletions(-)
create mode 100644 compat/fsmonitor/fsm-path-utils-darwin.c
create mode 100644 compat/fsmonitor/fsm-path-utils-win32.c
create mode 100644 fsmonitor-path-utils.h
diff --git a/Makefile b/Makefile
index 6bfb62cbe94..99a3d9e6322 100644
--- a/Makefile
+++ b/Makefile
@@ -2043,6 +2043,7 @@ endif
ifdef FSMONITOR_OS_SETTINGS
COMPAT_CFLAGS += -DHAVE_FSMONITOR_OS_SETTINGS
COMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_OS_SETTINGS).o
+ COMPAT_OBJS += compat/fsmonitor/fsm-path-utils-$(FSMONITOR_OS_SETTINGS).o
endif
ifeq ($(TCLTK_PATH),)
diff --git a/compat/fsmonitor/fsm-path-utils-darwin.c b/compat/fsmonitor/fsm-path-utils-darwin.c
new file mode 100644
index 00000000000..d46d7f13538
--- /dev/null
+++ b/compat/fsmonitor/fsm-path-utils-darwin.c
@@ -0,0 +1,43 @@
+#include "fsmonitor.h"
+#include "fsmonitor-path-utils.h"
+#include <sys/param.h>
+#include <sys/mount.h>
+
+int fsmonitor__get_fs_info(const char *path, struct fs_info *fs_info)
+{
+ struct statfs fs;
+ if (statfs(path, &fs) == -1) {
+ int saved_errno = errno;
+ trace_printf_key(&trace_fsmonitor, "statfs('%s') failed: %s",
+ path, strerror(saved_errno));
+ errno = saved_errno;
+ return -1;
+ }
+
+ trace_printf_key(&trace_fsmonitor,
+ "statfs('%s') [type 0x%08x][flags 0x%08x] '%s'",
+ path, fs.f_type, fs.f_flags, fs.f_fstypename);
+
+ if (!(fs.f_flags & MNT_LOCAL))
+ fs_info->is_remote = 1;
+ else
+ fs_info->is_remote = 0;
+
+ fs_info->typename = xstrdup(fs.f_fstypename);
+
+ trace_printf_key(&trace_fsmonitor,
+ "'%s' is_remote: %d",
+ path, fs_info->is_remote);
+ return 0;
+}
+
+int fsmonitor__is_fs_remote(const char *path)
+{
+ struct fs_info fs;
+ if (fsmonitor__get_fs_info(path, &fs))
+ return -1;
+
+ free(fs.typename);
+
+ return fs.is_remote;
+}
diff --git a/compat/fsmonitor/fsm-path-utils-win32.c b/compat/fsmonitor/fsm-path-utils-win32.c
new file mode 100644
index 00000000000..a90b8f7925b
--- /dev/null
+++ b/compat/fsmonitor/fsm-path-utils-win32.c
@@ -0,0 +1,128 @@
+#include "cache.h"
+#include "fsmonitor.h"
+#include "fsmonitor-path-utils.h"
+
+/*
+ * Check remote working directory protocol.
+ *
+ * Return -1 if client machine cannot get remote protocol information.
+ */
+static int check_remote_protocol(wchar_t *wpath)
+{
+ HANDLE h;
+ FILE_REMOTE_PROTOCOL_INFO proto_info;
+
+ h = CreateFileW(wpath, GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING,
+ FILE_FLAG_BACKUP_SEMANTICS, NULL);
+
+ if (h == INVALID_HANDLE_VALUE) {
+ error(_("[GLE %ld] unable to open for read '%ls'"),
+ GetLastError(), wpath);
+ return -1;
+ }
+
+ if (!GetFileInformationByHandleEx(h, FileRemoteProtocolInfo,
+ &proto_info, sizeof(proto_info))) {
+ error(_("[GLE %ld] unable to get protocol information for '%ls'"),
+ GetLastError(), wpath);
+ CloseHandle(h);
+ return -1;
+ }
+
+ CloseHandle(h);
+
+ trace_printf_key(&trace_fsmonitor,
+ "check_remote_protocol('%ls') remote protocol %#8.8lx",
+ wpath, proto_info.Protocol);
+
+ return 0;
+}
+
+/*
+ * Notes for testing:
+ *
+ * (a) Windows allows a network share to be mapped to a drive letter.
+ * (This is the normal method to access it.)
+ *
+ * $ NET USE Z: \\server\share
+ * $ git -C Z:/repo status
+ *
+ * (b) Windows allows a network share to be referenced WITHOUT mapping
+ * it to drive letter.
+ *
+ * $ NET USE \\server\share\dir
+ * $ git -C //server/share/repo status
+ *
+ * (c) Windows allows "SUBST" to create a fake drive mapping to an
+ * arbitrary path (which may be remote)
+ *
+ * $ SUBST Q: Z:\repo
+ * $ git -C Q:/ status
+ *
+ * (d) Windows allows a directory symlink to be created on a local
+ * file system that points to a remote repo.
+ *
+ * $ mklink /d ./link //server/share/repo
+ * $ git -C ./link status
+ */
+int fsmonitor__get_fs_info(const char *path, struct fs_info *fs_info)
+{
+ wchar_t wpath[MAX_PATH];
+ wchar_t wfullpath[MAX_PATH];
+ size_t wlen;
+ UINT driveType;
+
+ /*
+ * Do everything in wide chars because the drive letter might be
+ * a multi-byte sequence. See win32_has_dos_drive_prefix().
+ */
+ if (xutftowcs_path(wpath, path) < 0) {
+ return -1;
+ }
+
+ /*
+ * GetDriveTypeW() requires a final slash. We assume that the
+ * worktree pathname points to an actual directory.
+ */
+ wlen = wcslen(wpath);
+ if (wpath[wlen - 1] != L'\\' && wpath[wlen - 1] != L'/') {
+ wpath[wlen++] = L'\\';
+ wpath[wlen] = 0;
+ }
+
+ /*
+ * Normalize the path. If nothing else, this converts forward
+ * slashes to backslashes. This is essential to get GetDriveTypeW()
+ * correctly handle some UNC "\\server\share\..." paths.
+ */
+ if (!GetFullPathNameW(wpath, MAX_PATH, wfullpath, NULL)) {
+ return -1;
+ }
+
+ driveType = GetDriveTypeW(wfullpath);
+ trace_printf_key(&trace_fsmonitor,
+ "DriveType '%s' L'%ls' (%u)",
+ path, wfullpath, driveType);
+
+ if (driveType == DRIVE_REMOTE) {
+ fs_info->is_remote = 1;
+ if (check_remote_protocol(wfullpath) < 0)
+ return -1;
+ } else {
+ fs_info->is_remote = 0;
+ }
+
+ trace_printf_key(&trace_fsmonitor,
+ "'%s' is_remote: %d",
+ path, fs_info->is_remote);
+
+ return 0;
+}
+
+int fsmonitor__is_fs_remote(const char *path)
+{
+ struct fs_info fs;
+ if (fsmonitor__get_fs_info(path, &fs))
+ return -1;
+ return fs.is_remote;
+}
diff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c
index efc732c0f31..699f0b272e6 100644
--- a/compat/fsmonitor/fsm-settings-darwin.c
+++ b/compat/fsmonitor/fsm-settings-darwin.c
@@ -1,32 +1,10 @@
-#include "cache.h"
#include "config.h"
-#include "repository.h"
-#include "fsmonitor-settings.h"
#include "fsmonitor.h"
-#include <sys/param.h>
-#include <sys/mount.h>
+#include "fsmonitor-ipc.h"
+#include "fsmonitor-settings.h"
+#include "fsmonitor-path-utils.h"
-/*
- * [1] Remote working directories are problematic for FSMonitor.
- *
- * The underlying file system on the server machine and/or the remote
- * mount type (NFS, SAMBA, etc.) dictates whether notification events
- * are available at all to remote client machines.
- *
- * Kernel differences between the server and client machines also
- * dictate the how (buffering, frequency, de-dup) the events are
- * delivered to client machine processes.
- *
- * A client machine (such as a laptop) may choose to suspend/resume
- * and it is unclear (without lots of testing) whether the watcher can
- * resync after a resume. We might be able to treat this as a normal
- * "events were dropped by the kernel" event and do our normal "flush
- * and resync" --or-- we might need to close the existing (zombie?)
- * notification fd and create a new one.
- *
- * In theory, the above issues need to be addressed whether we are
- * using the Hook or IPC API.
- *
+ /*
* For the builtin FSMonitor, we create the Unix domain socket for the
* IPC in the .git directory. If the working directory is remote,
* then the socket will be created on the remote file system. This
@@ -38,42 +16,35 @@
* be taken to ensure that $HOME is actually local and not a managed
* file share.)
*
- * So (for now at least), mark remote working directories as
- * incompatible.
- *
- *
- * [2] FAT32 and NTFS working directories are problematic too.
+ * FAT32 and NTFS working directories are problematic too.
*
* The builtin FSMonitor uses a Unix domain socket in the .git
* directory for IPC. These Windows drive formats do not support
* Unix domain sockets, so mark them as incompatible for the daemon.
*
*/
-static enum fsmonitor_reason check_volume(struct repository *r)
+static enum fsmonitor_reason check_uds_volume(struct repository *r)
{
- struct statfs fs;
+ struct fs_info fs;
+ const char *ipc_path = fsmonitor_ipc__get_path();
+ struct strbuf path = STRBUF_INIT;
+ strbuf_add(&path, ipc_path, strlen(ipc_path));
- if (statfs(r->worktree, &fs) == -1) {
- int saved_errno = errno;
- trace_printf_key(&trace_fsmonitor, "statfs('%s') failed: %s",
- r->worktree, strerror(saved_errno));
- errno = saved_errno;
+ if (fsmonitor__get_fs_info(dirname(path.buf), &fs) == -1) {
+ strbuf_release(&path);
return FSMONITOR_REASON_ERROR;
}
- trace_printf_key(&trace_fsmonitor,
- "statfs('%s') [type 0x%08x][flags 0x%08x] '%s'",
- r->worktree, fs.f_type, fs.f_flags, fs.f_fstypename);
-
- if (!(fs.f_flags & MNT_LOCAL))
- return FSMONITOR_REASON_REMOTE;
+ strbuf_release(&path);
- if (!strcmp(fs.f_fstypename, "msdos")) /* aka FAT32 */
- return FSMONITOR_REASON_NOSOCKETS;
-
- if (!strcmp(fs.f_fstypename, "ntfs"))
+ if (fs.is_remote ||
+ !strcmp(fs.typename, "msdos") ||
+ !strcmp(fs.typename, "ntfs")) {
+ free(fs.typename);
return FSMONITOR_REASON_NOSOCKETS;
+ }
+ free(fs.typename);
return FSMONITOR_REASON_OK;
}
@@ -81,7 +52,7 @@ enum fsmonitor_reason fsm_os__incompatible(struct repository *r)
{
enum fsmonitor_reason reason;
- reason = check_volume(r);
+ reason = check_uds_volume(r);
if (reason != FSMONITOR_REASON_OK)
return reason;
diff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c
index e5ec5b0a9f7..d88b06ae610 100644
--- a/compat/fsmonitor/fsm-settings-win32.c
+++ b/compat/fsmonitor/fsm-settings-win32.c
@@ -1,8 +1,9 @@
#include "cache.h"
#include "config.h"
#include "repository.h"
-#include "fsmonitor-settings.h"
#include "fsmonitor.h"
+#include "fsmonitor-settings.h"
+#include "fsmonitor-path-utils.h"
/*
* VFS for Git is incompatible with FSMonitor.
@@ -24,171 +25,6 @@ static enum fsmonitor_reason check_vfs4git(struct repository *r)
return FSMONITOR_REASON_OK;
}
-/*
- * Check if monitoring remote working directories is allowed.
- *
- * By default, monitoring remote working directories is
- * disabled. Users may override this behavior in enviroments where
- * they have proper support.
- */
-static int check_config_allowremote(struct repository *r)
-{
- int allow;
-
- if (!repo_config_get_bool(r, "fsmonitor.allowremote", &allow))
- return allow;
-
- return -1; /* fsmonitor.allowremote not set */
-}
-
-/*
- * Check remote working directory protocol.
- *
- * Error if client machine cannot get remote protocol information.
- */
-static int check_remote_protocol(wchar_t *wpath)
-{
- HANDLE h;
- FILE_REMOTE_PROTOCOL_INFO proto_info;
-
- h = CreateFileW(wpath, GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING,
- FILE_FLAG_BACKUP_SEMANTICS, NULL);
-
- if (h == INVALID_HANDLE_VALUE) {
- error(_("[GLE %ld] unable to open for read '%ls'"),
- GetLastError(), wpath);
- return -1;
- }
-
- if (!GetFileInformationByHandleEx(h, FileRemoteProtocolInfo,
- &proto_info, sizeof(proto_info))) {
- error(_("[GLE %ld] unable to get protocol information for '%ls'"),
- GetLastError(), wpath);
- CloseHandle(h);
- return -1;
- }
-
- CloseHandle(h);
-
- trace_printf_key(&trace_fsmonitor,
- "check_remote_protocol('%ls') remote protocol %#8.8lx",
- wpath, proto_info.Protocol);
-
- return 0;
-}
-
-/*
- * Remote working directories are problematic for FSMonitor.
- *
- * The underlying file system on the server machine and/or the remote
- * mount type dictates whether notification events are available at
- * all to remote client machines.
- *
- * Kernel differences between the server and client machines also
- * dictate the how (buffering, frequency, de-dup) the events are
- * delivered to client machine processes.
- *
- * A client machine (such as a laptop) may choose to suspend/resume
- * and it is unclear (without lots of testing) whether the watcher can
- * resync after a resume. We might be able to treat this as a normal
- * "events were dropped by the kernel" event and do our normal "flush
- * and resync" --or-- we might need to close the existing (zombie?)
- * notification fd and create a new one.
- *
- * In theory, the above issues need to be addressed whether we are
- * using the Hook or IPC API.
- *
- * So (for now at least), mark remote working directories as
- * incompatible.
- *
- * Notes for testing:
- *
- * (a) Windows allows a network share to be mapped to a drive letter.
- * (This is the normal method to access it.)
- *
- * $ NET USE Z: \\server\share
- * $ git -C Z:/repo status
- *
- * (b) Windows allows a network share to be referenced WITHOUT mapping
- * it to drive letter.
- *
- * $ NET USE \\server\share\dir
- * $ git -C //server/share/repo status
- *
- * (c) Windows allows "SUBST" to create a fake drive mapping to an
- * arbitrary path (which may be remote)
- *
- * $ SUBST Q: Z:\repo
- * $ git -C Q:/ status
- *
- * (d) Windows allows a directory symlink to be created on a local
- * file system that points to a remote repo.
- *
- * $ mklink /d ./link //server/share/repo
- * $ git -C ./link status
- */
-static enum fsmonitor_reason check_remote(struct repository *r)
-{
- int ret;
- wchar_t wpath[MAX_PATH];
- wchar_t wfullpath[MAX_PATH];
- size_t wlen;
- UINT driveType;
-
- /*
- * Do everything in wide chars because the drive letter might be
- * a multi-byte sequence. See win32_has_dos_drive_prefix().
- */
- if (xutftowcs_path(wpath, r->worktree) < 0)
- return FSMONITOR_REASON_ERROR;
-
- /*
- * GetDriveTypeW() requires a final slash. We assume that the
- * worktree pathname points to an actual directory.
- */
- wlen = wcslen(wpath);
- if (wpath[wlen - 1] != L'\\' && wpath[wlen - 1] != L'/') {
- wpath[wlen++] = L'\\';
- wpath[wlen] = 0;
- }
-
- /*
- * Normalize the path. If nothing else, this converts forward
- * slashes to backslashes. This is essential to get GetDriveTypeW()
- * correctly handle some UNC "\\server\share\..." paths.
- */
- if (!GetFullPathNameW(wpath, MAX_PATH, wfullpath, NULL))
- return FSMONITOR_REASON_ERROR;
-
- driveType = GetDriveTypeW(wfullpath);
- trace_printf_key(&trace_fsmonitor,
- "DriveType '%s' L'%ls' (%u)",
- r->worktree, wfullpath, driveType);
-
- if (driveType == DRIVE_REMOTE) {
- trace_printf_key(&trace_fsmonitor,
- "check_remote('%s') true",
- r->worktree);
-
- ret = check_remote_protocol(wfullpath);
- if (ret < 0)
- return FSMONITOR_REASON_ERROR;
-
- switch (check_config_allowremote(r)) {
- case 0: /* config overrides and disables */
- return FSMONITOR_REASON_REMOTE;
- case 1: /* config overrides and enables */
- return FSMONITOR_REASON_OK;
- default:
- break; /* config has no opinion */
- }
-
- return FSMONITOR_REASON_REMOTE;
- }
-
- return FSMONITOR_REASON_OK;
-}
-
enum fsmonitor_reason fsm_os__incompatible(struct repository *r)
{
enum fsmonitor_reason reason;
@@ -197,9 +33,5 @@ enum fsmonitor_reason fsm_os__incompatible(struct repository *r)
if (reason != FSMONITOR_REASON_OK)
return reason;
- reason = check_remote(r);
- if (reason != FSMONITOR_REASON_OK)
- return reason;
-
return FSMONITOR_REASON_OK;
}
diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt
index ea2a531be87..5482a04b3ce 100644
--- a/contrib/buildsystems/CMakeLists.txt
+++ b/contrib/buildsystems/CMakeLists.txt
@@ -308,6 +308,7 @@ if(SUPPORTS_SIMPLE_IPC)
add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)
list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)
list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-win32.c)
+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-win32.c)
add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)
list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c)
@@ -315,6 +316,7 @@ if(SUPPORTS_SIMPLE_IPC)
add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)
list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)
list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-darwin.c)
+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-darwin.c)
add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)
list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c)
diff --git a/fsmonitor-path-utils.h b/fsmonitor-path-utils.h
new file mode 100644
index 00000000000..41edf5b934f
--- /dev/null
+++ b/fsmonitor-path-utils.h
@@ -0,0 +1,26 @@
+#ifndef FSM_PATH_UTILS_H
+#define FSM_PATH_UTILS_H
+
+struct fs_info {
+ int is_remote;
+ char *typename;
+};
+
+/*
+ * Get some basic filesystem informtion for the given path
+ *
+ * The caller owns the storage that is occupied by fs_info and
+ * is responsible for releasing it.
+ *
+ * Returns -1 on error, zero otherwise.
+ */
+int fsmonitor__get_fs_info(const char *path, struct fs_info *fs_info);
+
+/*
+ * Determines if the filesystem that path resides on is remote.
+ *
+ * Returns -1 on error, 0 if not remote, 1 if remote.
+ */
+int fsmonitor__is_fs_remote(const char *path);
+
+#endif
diff --git a/fsmonitor-settings.c b/fsmonitor-settings.c
index 464424a1e92..d288cbad479 100644
--- a/fsmonitor-settings.c
+++ b/fsmonitor-settings.c
@@ -2,6 +2,7 @@
#include "config.h"
#include "repository.h"
#include "fsmonitor-settings.h"
+#include "fsmonitor-path-utils.h"
/*
* We keep this structure defintion private and have getters
@@ -13,6 +14,52 @@ struct fsmonitor_settings {
char *hook_path;
};
+/*
+ * Remote working directories are problematic for FSMonitor.
+ *
+ * The underlying file system on the server machine and/or the remote
+ * mount type dictates whether notification events are available at
+ * all to remote client machines.
+ *
+ * Kernel differences between the server and client machines also
+ * dictate the how (buffering, frequency, de-dup) the events are
+ * delivered to client machine processes.
+ *
+ * A client machine (such as a laptop) may choose to suspend/resume
+ * and it is unclear (without lots of testing) whether the watcher can
+ * resync after a resume. We might be able to treat this as a normal
+ * "events were dropped by the kernel" event and do our normal "flush
+ * and resync" --or-- we might need to close the existing (zombie?)
+ * notification fd and create a new one.
+ *
+ * In theory, the above issues need to be addressed whether we are
+ * using the Hook or IPC API.
+ *
+ * So (for now at least), mark remote working directories as
+ * incompatible unless 'fsmonitor.allowRemote' is true.
+ *
+ */
+#ifdef HAVE_FSMONITOR_OS_SETTINGS
+static enum fsmonitor_reason check_remote(struct repository *r)
+{
+ int allow_remote = -1; /* -1 unset, 0 not allowed, 1 allowed */
+ int is_remote = fsmonitor__is_fs_remote(r->worktree);
+
+ switch (is_remote) {
+ case 0:
+ return FSMONITOR_REASON_OK;
+ case 1:
+ repo_config_get_bool(r, "fsmonitor.allowremote", &allow_remote);
+ if (allow_remote < 1)
+ return FSMONITOR_REASON_REMOTE;
+ else
+ return FSMONITOR_REASON_OK;
+ default:
+ return FSMONITOR_REASON_ERROR;
+ }
+}
+#endif
+
static enum fsmonitor_reason check_for_incompatible(struct repository *r)
{
if (!r->worktree) {
@@ -27,6 +74,9 @@ static enum fsmonitor_reason check_for_incompatible(struct repository *r)
{
enum fsmonitor_reason reason;
+ reason = check_remote(r);
+ if (reason != FSMONITOR_REASON_OK)
+ return reason;
reason = fsm_os__incompatible(r);
if (reason != FSMONITOR_REASON_OK)
return reason;
--
gitgitgadget
^ permalink raw reply related [flat|nested] 89+ messages in thread* [PATCH v2 02/12] fsmonitor: relocate socket file if .git directory is remote
2022-10-14 21:45 ` [PATCH v2 " Eric DeCosta via GitGitGadget
2022-10-14 21:45 ` [PATCH v2 01/12] fsmonitor: refactor filesystem checks to common interface Eric DeCosta via GitGitGadget
@ 2022-10-14 21:45 ` Eric DeCosta via GitGitGadget
2022-10-18 12:12 ` Ævar Arnfjörð Bjarmason
2022-10-14 21:45 ` [PATCH v2 03/12] fsmonitor: avoid socket location check if using hook Eric DeCosta via GitGitGadget
` (12 subsequent siblings)
14 siblings, 1 reply; 89+ messages in thread
From: Eric DeCosta via GitGitGadget @ 2022-10-14 21:45 UTC (permalink / raw)
To: git
Cc: Eric Sunshine, Ævar Arnfjörð Bjarmason,
Eric DeCosta, Eric DeCosta
From: Eric DeCosta <edecosta@mathworks.com>
If the .git directory is on a remote filesystem, create the socket
file in 'fsmonitor.socketDir' if it is defined, else create it in $HOME.
Signed-off-by: Eric DeCosta <edecosta@mathworks.com>
---
Makefile | 1 +
builtin/fsmonitor--daemon.c | 3 +-
compat/fsmonitor/fsm-ipc-darwin.c | 52 ++++++++++++++++++++++++++
compat/fsmonitor/fsm-ipc-win32.c | 9 +++++
compat/fsmonitor/fsm-settings-darwin.c | 2 +-
contrib/buildsystems/CMakeLists.txt | 2 +
fsmonitor-ipc.c | 18 ++++-----
fsmonitor-ipc.h | 4 +-
8 files changed, 78 insertions(+), 13 deletions(-)
create mode 100644 compat/fsmonitor/fsm-ipc-darwin.c
create mode 100644 compat/fsmonitor/fsm-ipc-win32.c
diff --git a/Makefile b/Makefile
index 99a3d9e6322..9553d590824 100644
--- a/Makefile
+++ b/Makefile
@@ -2038,6 +2038,7 @@ ifdef FSMONITOR_DAEMON_BACKEND
COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND
COMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o
COMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_BACKEND).o
+ COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_DAEMON_BACKEND).o
endif
ifdef FSMONITOR_OS_SETTINGS
diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c
index c69da93eceb..6e417846d8b 100644
--- a/builtin/fsmonitor--daemon.c
+++ b/builtin/fsmonitor--daemon.c
@@ -1343,7 +1343,8 @@ static int fsmonitor_run_daemon(void)
* directory.)
*/
strbuf_init(&state.path_ipc, 0);
- strbuf_addstr(&state.path_ipc, absolute_path(fsmonitor_ipc__get_path()));
+ strbuf_addstr(&state.path_ipc,
+ absolute_path(fsmonitor_ipc__get_path(the_repository)));
/*
* Confirm that we can create platform-specific resources for the
diff --git a/compat/fsmonitor/fsm-ipc-darwin.c b/compat/fsmonitor/fsm-ipc-darwin.c
new file mode 100644
index 00000000000..ce843d63348
--- /dev/null
+++ b/compat/fsmonitor/fsm-ipc-darwin.c
@@ -0,0 +1,52 @@
+#include "cache.h"
+#include "config.h"
+#include "strbuf.h"
+#include "fsmonitor.h"
+#include "fsmonitor-ipc.h"
+#include "fsmonitor-path-utils.h"
+
+static GIT_PATH_FUNC(fsmonitor_ipc__get_default_path, "fsmonitor--daemon.ipc")
+
+const char *fsmonitor_ipc__get_path(struct repository *r)
+{
+ static const char *ipc_path = NULL;
+ SHA_CTX sha1ctx;
+ char *sock_dir = NULL;
+ struct strbuf ipc_file = STRBUF_INIT;
+ unsigned char hash[SHA_DIGEST_LENGTH];
+
+ if (!r)
+ BUG("No repository passed into fsmonitor_ipc__get_path");
+
+ if (ipc_path)
+ return ipc_path;
+
+
+ /* By default the socket file is created in the .git directory */
+ if (fsmonitor__is_fs_remote(r->gitdir) < 1) {
+ ipc_path = fsmonitor_ipc__get_default_path();
+ return ipc_path;
+ }
+
+ SHA1_Init(&sha1ctx);
+ SHA1_Update(&sha1ctx, r->worktree, strlen(r->worktree));
+ SHA1_Final(hash, &sha1ctx);
+
+ repo_config_get_string(r, "fsmonitor.socketdir", &sock_dir);
+
+ /* Create the socket file in either socketDir or $HOME */
+ if (sock_dir && *sock_dir) {
+ strbuf_addf(&ipc_file, "%s/.git-fsmonitor-%s",
+ sock_dir, hash_to_hex(hash));
+ } else {
+ strbuf_addf(&ipc_file, "~/.git-fsmonitor-%s", hash_to_hex(hash));
+ }
+ free(sock_dir);
+
+ ipc_path = interpolate_path(ipc_file.buf, 1);
+ if (!ipc_path)
+ die(_("Invalid path: %s"), ipc_file.buf);
+
+ strbuf_release(&ipc_file);
+ return ipc_path;
+}
diff --git a/compat/fsmonitor/fsm-ipc-win32.c b/compat/fsmonitor/fsm-ipc-win32.c
new file mode 100644
index 00000000000..e08c505c148
--- /dev/null
+++ b/compat/fsmonitor/fsm-ipc-win32.c
@@ -0,0 +1,9 @@
+#include "config.h"
+#include "fsmonitor-ipc.h"
+
+const char *fsmonitor_ipc__get_path(struct repository *r) {
+ static char *ret;
+ if (!ret)
+ ret = git_pathdup("fsmonitor--daemon.ipc");
+ return ret;
+}
diff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c
index 699f0b272e6..7241c4c22c9 100644
--- a/compat/fsmonitor/fsm-settings-darwin.c
+++ b/compat/fsmonitor/fsm-settings-darwin.c
@@ -26,7 +26,7 @@
static enum fsmonitor_reason check_uds_volume(struct repository *r)
{
struct fs_info fs;
- const char *ipc_path = fsmonitor_ipc__get_path();
+ const char *ipc_path = fsmonitor_ipc__get_path(r);
struct strbuf path = STRBUF_INIT;
strbuf_add(&path, ipc_path, strlen(ipc_path));
diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt
index 5482a04b3ce..787738e6fa3 100644
--- a/contrib/buildsystems/CMakeLists.txt
+++ b/contrib/buildsystems/CMakeLists.txt
@@ -308,6 +308,7 @@ if(SUPPORTS_SIMPLE_IPC)
add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)
list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)
list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-win32.c)
+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-win32.c)
list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-win32.c)
add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)
@@ -316,6 +317,7 @@ if(SUPPORTS_SIMPLE_IPC)
add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)
list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)
list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-darwin.c)
+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-darwin.c)
list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-darwin.c)
add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)
diff --git a/fsmonitor-ipc.c b/fsmonitor-ipc.c
index 789e7397baa..c0f42301c84 100644
--- a/fsmonitor-ipc.c
+++ b/fsmonitor-ipc.c
@@ -18,7 +18,7 @@ int fsmonitor_ipc__is_supported(void)
return 0;
}
-const char *fsmonitor_ipc__get_path(void)
+const char *fsmonitor_ipc__get_path(struct repository *r)
{
return NULL;
}
@@ -47,11 +47,9 @@ int fsmonitor_ipc__is_supported(void)
return 1;
}
-GIT_PATH_FUNC(fsmonitor_ipc__get_path, "fsmonitor--daemon.ipc")
-
enum ipc_active_state fsmonitor_ipc__get_state(void)
{
- return ipc_get_active_state(fsmonitor_ipc__get_path());
+ return ipc_get_active_state(fsmonitor_ipc__get_path(the_repository));
}
static int spawn_daemon(void)
@@ -81,8 +79,8 @@ int fsmonitor_ipc__send_query(const char *since_token,
trace2_data_string("fsm_client", NULL, "query/command", tok);
try_again:
- state = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options,
- &connection);
+ state = ipc_client_try_connect(fsmonitor_ipc__get_path(the_repository),
+ &options, &connection);
switch (state) {
case IPC_STATE__LISTENING:
@@ -117,13 +115,13 @@ try_again:
case IPC_STATE__INVALID_PATH:
ret = error(_("fsmonitor_ipc__send_query: invalid path '%s'"),
- fsmonitor_ipc__get_path());
+ fsmonitor_ipc__get_path(the_repository));
goto done;
case IPC_STATE__OTHER_ERROR:
default:
ret = error(_("fsmonitor_ipc__send_query: unspecified error on '%s'"),
- fsmonitor_ipc__get_path());
+ fsmonitor_ipc__get_path(the_repository));
goto done;
}
@@ -149,8 +147,8 @@ int fsmonitor_ipc__send_command(const char *command,
options.wait_if_busy = 1;
options.wait_if_not_found = 0;
- state = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options,
- &connection);
+ state = ipc_client_try_connect(fsmonitor_ipc__get_path(the_repository),
+ &options, &connection);
if (state != IPC_STATE__LISTENING) {
die(_("fsmonitor--daemon is not running"));
return -1;
diff --git a/fsmonitor-ipc.h b/fsmonitor-ipc.h
index b6a7067c3af..8b489da762b 100644
--- a/fsmonitor-ipc.h
+++ b/fsmonitor-ipc.h
@@ -3,6 +3,8 @@
#include "simple-ipc.h"
+struct repository;
+
/*
* Returns true if built-in file system monitor daemon is defined
* for this platform.
@@ -16,7 +18,7 @@ int fsmonitor_ipc__is_supported(void);
*
* Returns NULL if the daemon is not supported on this platform.
*/
-const char *fsmonitor_ipc__get_path(void);
+const char *fsmonitor_ipc__get_path(struct repository *r);
/*
* Try to determine whether there is a `git-fsmonitor--daemon` process
--
gitgitgadget
^ permalink raw reply related [flat|nested] 89+ messages in thread* Re: [PATCH v2 02/12] fsmonitor: relocate socket file if .git directory is remote
2022-10-14 21:45 ` [PATCH v2 02/12] fsmonitor: relocate socket file if .git directory is remote Eric DeCosta via GitGitGadget
@ 2022-10-18 12:12 ` Ævar Arnfjörð Bjarmason
0 siblings, 0 replies; 89+ messages in thread
From: Ævar Arnfjörð Bjarmason @ 2022-10-18 12:12 UTC (permalink / raw)
To: Eric DeCosta via GitGitGadget; +Cc: git, Eric Sunshine, Eric DeCosta
On Fri, Oct 14 2022, Eric DeCosta via GitGitGadget wrote:
> From: Eric DeCosta <edecosta@mathworks.com>
> [..]
> +const char *fsmonitor_ipc__get_path(struct repository *r)
> +{
> + static const char *ipc_path = NULL;
Urm, I see some of this interface made it into master already with
6beb2688d33 (fsmonitor: relocate socket file if .git directory is
remote, 2022-10-04), but this is just weird. We already have "struct
fsmonitor_settings" which "lazily loads" these various settings.
So e.g. in refresh_fsmonitor() we might get the hook path, which we
"lazy load" and cache there.
And here we have an extra layer of another type of lazy loading, and
this will e.g. be called from fsmonitor-settings.c.
Is there really a good reason for why the relevant codepaths can't
either not-lazy-load this (e.g. when we get the socket), or why we can't
just use fsmonitor-settings.c's existing caching?
Just wondering...
> + repo_config_get_string(r, "fsmonitor.socketdir", &sock_dir);
> +
> + /* Create the socket file in either socketDir or $HOME */
> + if (sock_dir && *sock_dir) {
> + strbuf_addf(&ipc_file, "%s/.git-fsmonitor-%s",
> + sock_dir, hash_to_hex(hash));
> + } else {
> + strbuf_addf(&ipc_file, "~/.git-fsmonitor-%s", hash_to_hex(hash));
> + }
> + free(sock_dir);
Instead of this init sock_dir to NULL & then check if it's non-NULL
etc. Just do:
char *sock_dir;
if (!repo_config_get_string_tmp(r, "fsmonitor.socketdir", &sock_dir) &&
*sock_dir)
...
else
...
Note also the *_tmp(), then you won't have to free() it.
> +
> + ipc_path = interpolate_path(ipc_file.buf, 1);
> + if (!ipc_path)
> + die(_("Invalid path: %s"), ipc_file.buf);
> +
> + strbuf_release(&ipc_file);
> + return ipc_path;
Why can't we just do something more simpler like this in the *nix case?
I think because you're creating this socket for a NFS-mounted repo, so
of course we might have multiple clients.
But you also seem to be making it "unique" by hashing r->worktree, which
is the path to the repo work tree. At least in my use-cases for NFS
mounted repos (which have been limited) all of those paths would be the
same for me across my systems, because I'd have them at
/home/avar/g/some-git-tree/...
In builtin/gc.c we create a "unique for system" lock file using
xgethostname(), isn't that a lot simpler & closer to what you actually
want?
In any case, some details about motivations, why etc. are sorely missing
in the commit message. We can see *what* you're doing, but not *why*, or
what other simpler solutions etc. might have been tried & dismissed...
^ permalink raw reply [flat|nested] 89+ messages in thread
* [PATCH v2 03/12] fsmonitor: avoid socket location check if using hook
2022-10-14 21:45 ` [PATCH v2 " Eric DeCosta via GitGitGadget
2022-10-14 21:45 ` [PATCH v2 01/12] fsmonitor: refactor filesystem checks to common interface Eric DeCosta via GitGitGadget
2022-10-14 21:45 ` [PATCH v2 02/12] fsmonitor: relocate socket file if .git directory is remote Eric DeCosta via GitGitGadget
@ 2022-10-14 21:45 ` Eric DeCosta via GitGitGadget
2022-10-14 21:45 ` [PATCH v2 04/12] fsmonitor: deal with synthetic firmlinks on macOS Eric DeCosta via GitGitGadget
` (11 subsequent siblings)
14 siblings, 0 replies; 89+ messages in thread
From: Eric DeCosta via GitGitGadget @ 2022-10-14 21:45 UTC (permalink / raw)
To: git
Cc: Eric Sunshine, Ævar Arnfjörð Bjarmason,
Eric DeCosta, Eric DeCosta
From: Eric DeCosta <edecosta@mathworks.com>
If monitoring is done via fsmonitor hook rather than IPC there is no
need to check if the location of the Unix Domain socket (UDS) file is
on a remote filesystem.
Signed-off-by: Eric DeCosta <edecosta@mathworks.com>
---
compat/fsmonitor/fsm-settings-darwin.c | 10 ++++++----
compat/fsmonitor/fsm-settings-win32.c | 2 +-
fsmonitor-settings.c | 8 ++++----
fsmonitor-settings.h | 2 +-
4 files changed, 12 insertions(+), 10 deletions(-)
diff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c
index 7241c4c22c9..6abbc7af3ab 100644
--- a/compat/fsmonitor/fsm-settings-darwin.c
+++ b/compat/fsmonitor/fsm-settings-darwin.c
@@ -48,13 +48,15 @@ static enum fsmonitor_reason check_uds_volume(struct repository *r)
return FSMONITOR_REASON_OK;
}
-enum fsmonitor_reason fsm_os__incompatible(struct repository *r)
+enum fsmonitor_reason fsm_os__incompatible(struct repository *r, int ipc)
{
enum fsmonitor_reason reason;
- reason = check_uds_volume(r);
- if (reason != FSMONITOR_REASON_OK)
- return reason;
+ if (ipc) {
+ reason = check_uds_volume(r);
+ if (reason != FSMONITOR_REASON_OK)
+ return reason;
+ }
return FSMONITOR_REASON_OK;
}
diff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c
index d88b06ae610..a8af31b71de 100644
--- a/compat/fsmonitor/fsm-settings-win32.c
+++ b/compat/fsmonitor/fsm-settings-win32.c
@@ -25,7 +25,7 @@ static enum fsmonitor_reason check_vfs4git(struct repository *r)
return FSMONITOR_REASON_OK;
}
-enum fsmonitor_reason fsm_os__incompatible(struct repository *r)
+enum fsmonitor_reason fsm_os__incompatible(struct repository *r, int ipc)
{
enum fsmonitor_reason reason;
diff --git a/fsmonitor-settings.c b/fsmonitor-settings.c
index d288cbad479..531a1b6f956 100644
--- a/fsmonitor-settings.c
+++ b/fsmonitor-settings.c
@@ -60,7 +60,7 @@ static enum fsmonitor_reason check_remote(struct repository *r)
}
#endif
-static enum fsmonitor_reason check_for_incompatible(struct repository *r)
+static enum fsmonitor_reason check_for_incompatible(struct repository *r, int ipc)
{
if (!r->worktree) {
/*
@@ -77,7 +77,7 @@ static enum fsmonitor_reason check_for_incompatible(struct repository *r)
reason = check_remote(r);
if (reason != FSMONITOR_REASON_OK)
return reason;
- reason = fsm_os__incompatible(r);
+ reason = fsm_os__incompatible(r, ipc);
if (reason != FSMONITOR_REASON_OK)
return reason;
}
@@ -162,7 +162,7 @@ const char *fsm_settings__get_hook_path(struct repository *r)
void fsm_settings__set_ipc(struct repository *r)
{
- enum fsmonitor_reason reason = check_for_incompatible(r);
+ enum fsmonitor_reason reason = check_for_incompatible(r, 1);
if (reason != FSMONITOR_REASON_OK) {
fsm_settings__set_incompatible(r, reason);
@@ -185,7 +185,7 @@ void fsm_settings__set_ipc(struct repository *r)
void fsm_settings__set_hook(struct repository *r, const char *path)
{
- enum fsmonitor_reason reason = check_for_incompatible(r);
+ enum fsmonitor_reason reason = check_for_incompatible(r, 0);
if (reason != FSMONITOR_REASON_OK) {
fsm_settings__set_incompatible(r, reason);
diff --git a/fsmonitor-settings.h b/fsmonitor-settings.h
index d9c2605197f..0721617b95a 100644
--- a/fsmonitor-settings.h
+++ b/fsmonitor-settings.h
@@ -48,7 +48,7 @@ struct fsmonitor_settings;
* fsm_os__* routines should considered private to fsm_settings__
* routines.
*/
-enum fsmonitor_reason fsm_os__incompatible(struct repository *r);
+enum fsmonitor_reason fsm_os__incompatible(struct repository *r, int ipc);
#endif /* HAVE_FSMONITOR_OS_SETTINGS */
#endif /* FSMONITOR_SETTINGS_H */
--
gitgitgadget
^ permalink raw reply related [flat|nested] 89+ messages in thread* [PATCH v2 04/12] fsmonitor: deal with synthetic firmlinks on macOS
2022-10-14 21:45 ` [PATCH v2 " Eric DeCosta via GitGitGadget
` (2 preceding siblings ...)
2022-10-14 21:45 ` [PATCH v2 03/12] fsmonitor: avoid socket location check if using hook Eric DeCosta via GitGitGadget
@ 2022-10-14 21:45 ` Eric DeCosta via GitGitGadget
2022-10-14 21:45 ` [PATCH v2 05/12] fsmonitor: check for compatability before communicating with fsmonitor Eric DeCosta via GitGitGadget
` (10 subsequent siblings)
14 siblings, 0 replies; 89+ messages in thread
From: Eric DeCosta via GitGitGadget @ 2022-10-14 21:45 UTC (permalink / raw)
To: git
Cc: Eric Sunshine, Ævar Arnfjörð Bjarmason,
Eric DeCosta, Eric DeCosta
From: Eric DeCosta <edecosta@mathworks.com>
Starting with macOS 10.15 (Catalina), Apple introduced a new feature
called 'firmlinks' in order to separate the boot volume into two
volumes, one read-only and one writable but still present them to the
user as a single volume. Along with this change, Apple removed the
ability to create symlinks in the root directory and replaced them with
'synthetic firmlinks'. See 'man synthetic.conf'
When FSEevents reports the path of changed files, if the path involves
a synthetic firmlink, the path is reported from the point of the
synthetic firmlink and not the real path. For example:
Real path:
/System/Volumes/Data/network/working/directory/foo.txt
Synthetic firmlink:
/network -> /System/Volumes/Data/network
FSEvents path:
/network/working/directory/foo.txt
This causes the FSEvents path to not match against the worktree
directory.
There are several ways in which synthetic firmlinks can be created:
they can be defined in /etc/synthetic.conf, the automounter can create
them, and there may be other means. Simply reading /etc/synthetic.conf
is insufficient. No matter what process creates synthetic firmlinks,
they all get created in the root directory.
Therefore, in order to deal with synthetic firmlinks, the root directory
is scanned and the first possible synthetic firmink that, when resolved,
is a prefix of the worktree is used to map FSEvents paths to worktree
paths.
Signed-off-by: Eric DeCosta <edecosta@mathworks.com>
---
builtin/fsmonitor--daemon.c | 8 +++
compat/fsmonitor/fsm-listen-darwin.c | 14 +++-
compat/fsmonitor/fsm-path-utils-darwin.c | 92 ++++++++++++++++++++++++
compat/fsmonitor/fsm-path-utils-win32.c | 17 +++++
fsmonitor--daemon.h | 3 +
fsmonitor-path-utils.h | 36 +++++++++-
6 files changed, 167 insertions(+), 3 deletions(-)
diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c
index 6e417846d8b..6f30a4f93a7 100644
--- a/builtin/fsmonitor--daemon.c
+++ b/builtin/fsmonitor--daemon.c
@@ -3,6 +3,7 @@
#include "parse-options.h"
#include "fsmonitor.h"
#include "fsmonitor-ipc.h"
+#include "fsmonitor-path-utils.h"
#include "compat/fsmonitor/fsm-health.h"
#include "compat/fsmonitor/fsm-listen.h"
#include "fsmonitor--daemon.h"
@@ -1282,6 +1283,11 @@ static int fsmonitor_run_daemon(void)
strbuf_addstr(&state.path_worktree_watch, absolute_path(get_git_work_tree()));
state.nr_paths_watching = 1;
+ strbuf_init(&state.alias.alias, 0);
+ strbuf_init(&state.alias.points_to, 0);
+ if ((err = fsmonitor__get_alias(state.path_worktree_watch.buf, &state.alias)))
+ goto done;
+
/*
* We create and delete cookie files somewhere inside the .git
* directory to help us keep sync with the file system. If
@@ -1391,6 +1397,8 @@ done:
strbuf_release(&state.path_gitdir_watch);
strbuf_release(&state.path_cookie_prefix);
strbuf_release(&state.path_ipc);
+ strbuf_release(&state.alias.alias);
+ strbuf_release(&state.alias.points_to);
return err;
}
diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c
index 8e208e8289e..daeee4e465c 100644
--- a/compat/fsmonitor/fsm-listen-darwin.c
+++ b/compat/fsmonitor/fsm-listen-darwin.c
@@ -26,6 +26,7 @@
#include "fsmonitor.h"
#include "fsm-listen.h"
#include "fsmonitor--daemon.h"
+#include "fsmonitor-path-utils.h"
struct fsm_listen_data
{
@@ -198,8 +199,9 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,
struct string_list cookie_list = STRING_LIST_INIT_DUP;
const char *path_k;
const char *slash;
- int k;
+ char *resolved = NULL;
struct strbuf tmp = STRBUF_INIT;
+ int k;
/*
* Build a list of all filesystem changes into a private/local
@@ -209,7 +211,12 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,
/*
* On Mac, we receive an array of absolute paths.
*/
- path_k = paths[k];
+ free(resolved);
+ resolved = fsmonitor__resolve_alias(paths[k], &state->alias);
+ if (resolved)
+ path_k = resolved;
+ else
+ path_k = paths[k];
/*
* If you want to debug FSEvents, log them to GIT_TRACE_FSMONITOR.
@@ -238,6 +245,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,
fsmonitor_force_resync(state);
fsmonitor_batch__free_list(batch);
string_list_clear(&cookie_list, 0);
+ batch = NULL;
/*
* We assume that any events that we received
@@ -360,12 +368,14 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef,
}
}
+ free(resolved);
fsmonitor_publish(state, batch, &cookie_list);
string_list_clear(&cookie_list, 0);
strbuf_release(&tmp);
return;
force_shutdown:
+ free(resolved);
fsmonitor_batch__free_list(batch);
string_list_clear(&cookie_list, 0);
diff --git a/compat/fsmonitor/fsm-path-utils-darwin.c b/compat/fsmonitor/fsm-path-utils-darwin.c
index d46d7f13538..ce5a8febe09 100644
--- a/compat/fsmonitor/fsm-path-utils-darwin.c
+++ b/compat/fsmonitor/fsm-path-utils-darwin.c
@@ -1,5 +1,8 @@
#include "fsmonitor.h"
#include "fsmonitor-path-utils.h"
+#include <dirent.h>
+#include <errno.h>
+#include <fcntl.h>
#include <sys/param.h>
#include <sys/mount.h>
@@ -41,3 +44,92 @@ int fsmonitor__is_fs_remote(const char *path)
return fs.is_remote;
}
+
+/*
+ * Scan the root directory for synthetic firmlinks that when resolved
+ * are a prefix of the path, stopping at the first one found.
+ *
+ * Some information about firmlinks and synthetic firmlinks:
+ * https://eclecticlight.co/2020/01/23/catalina-boot-volumes/
+ *
+ * macOS no longer allows symlinks in the root directory; any link found
+ * there is therefore a synthetic firmlink.
+ *
+ * If this function gets called often, will want to cache all the firmlink
+ * information, but for now there is only one caller of this function.
+ *
+ * If there is more than one alias for the path, that is another
+ * matter altogether.
+ */
+int fsmonitor__get_alias(const char *path, struct alias_info *info)
+{
+ DIR *dir;
+ int retval = -1;
+ const char *const root = "/";
+ struct stat st;
+ struct dirent *de;
+ struct strbuf alias;
+ struct strbuf points_to = STRBUF_INIT;
+
+ dir = opendir(root);
+ if (!dir)
+ return error_errno(_("opendir('%s') failed"), root);
+
+ strbuf_init(&alias, 256);
+
+ while ((de = readdir(dir)) != NULL) {
+ strbuf_reset(&alias);
+ strbuf_addf(&alias, "%s%s", root, de->d_name);
+
+ if (lstat(alias.buf, &st) < 0) {
+ error_errno(_("lstat('%s') failed"), alias.buf);
+ goto done;
+ }
+
+ if (!S_ISLNK(st.st_mode))
+ continue;
+
+ if (strbuf_readlink(&points_to, alias.buf, st.st_size) < 0) {
+ error_errno(_("strbuf_readlink('%s') failed"), alias.buf);
+ goto done;
+ }
+
+ if (!strncmp(points_to.buf, path, points_to.len) &&
+ (path[points_to.len] == '/')) {
+ strbuf_addbuf(&info->alias, &alias);
+ strbuf_addbuf(&info->points_to, &points_to);
+ trace_printf_key(&trace_fsmonitor,
+ "Found alias for '%s' : '%s' -> '%s'",
+ path, info->alias.buf, info->points_to.buf);
+ retval = 0;
+ goto done;
+ }
+ }
+ retval = 0; /* no alias */
+
+done:
+ strbuf_release(&alias);
+ strbuf_release(&points_to);
+ if (closedir(dir) < 0)
+ return error_errno(_("closedir('%s') failed"), root);
+ return retval;
+}
+
+char *fsmonitor__resolve_alias(const char *path,
+ const struct alias_info *info)
+{
+ if (!info->alias.len)
+ return NULL;
+
+ if ((!strncmp(info->alias.buf, path, info->alias.len))
+ && path[info->alias.len] == '/') {
+ struct strbuf tmp = STRBUF_INIT;
+ const char *remainder = path + info->alias.len;
+
+ strbuf_addbuf(&tmp, &info->points_to);
+ strbuf_add(&tmp, remainder, strlen(remainder));
+ return strbuf_detach(&tmp, NULL);
+ }
+
+ return NULL;
+}
diff --git a/compat/fsmonitor/fsm-path-utils-win32.c b/compat/fsmonitor/fsm-path-utils-win32.c
index a90b8f7925b..0d95bbb416f 100644
--- a/compat/fsmonitor/fsm-path-utils-win32.c
+++ b/compat/fsmonitor/fsm-path-utils-win32.c
@@ -126,3 +126,20 @@ int fsmonitor__is_fs_remote(const char *path)
return -1;
return fs.is_remote;
}
+
+/*
+ * No-op for now.
+ */
+int fsmonitor__get_alias(const char *path, struct alias_info *info)
+{
+ return 0;
+}
+
+/*
+ * No-op for now.
+ */
+char *fsmonitor__resolve_alias(const char *path,
+ const struct alias_info *info)
+{
+ return NULL;
+}
diff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h
index 2102a5c9ff5..e24838f9a86 100644
--- a/fsmonitor--daemon.h
+++ b/fsmonitor--daemon.h
@@ -8,6 +8,7 @@
#include "run-command.h"
#include "simple-ipc.h"
#include "thread-utils.h"
+#include "fsmonitor-path-utils.h"
struct fsmonitor_batch;
struct fsmonitor_token_data;
@@ -43,6 +44,7 @@ struct fsmonitor_daemon_state {
struct strbuf path_worktree_watch;
struct strbuf path_gitdir_watch;
+ struct alias_info alias;
int nr_paths_watching;
struct fsmonitor_token_data *current_token_data;
@@ -59,6 +61,7 @@ struct fsmonitor_daemon_state {
struct ipc_server_data *ipc_server_data;
struct strbuf path_ipc;
+
};
/*
diff --git a/fsmonitor-path-utils.h b/fsmonitor-path-utils.h
index 41edf5b934f..5bfdfb81c14 100644
--- a/fsmonitor-path-utils.h
+++ b/fsmonitor-path-utils.h
@@ -1,13 +1,21 @@
#ifndef FSM_PATH_UTILS_H
#define FSM_PATH_UTILS_H
+#include "strbuf.h"
+
+struct alias_info
+{
+ struct strbuf alias;
+ struct strbuf points_to;
+};
+
struct fs_info {
int is_remote;
char *typename;
};
/*
- * Get some basic filesystem informtion for the given path
+ * Get some basic filesystem information for the given path
*
* The caller owns the storage that is occupied by fs_info and
* is responsible for releasing it.
@@ -23,4 +31,30 @@ int fsmonitor__get_fs_info(const char *path, struct fs_info *fs_info);
*/
int fsmonitor__is_fs_remote(const char *path);
+/*
+ * Get the alias in given path, if any.
+ *
+ * Sets alias to the first alias that matches any part of the path.
+ *
+ * If an alias is found, info.alias and info.points_to are set to the
+ * found mapping.
+ *
+ * Returns -1 on error, 0 otherwise.
+ *
+ * The caller owns the storage that is occupied by info.alias and
+ * info.points_to and is responsible for releasing it.
+ */
+int fsmonitor__get_alias(const char *path, struct alias_info *info);
+
+/*
+ * Resolve the path against the given alias.
+ *
+ * Returns the resolved path if there is one, NULL otherwise.
+ *
+ * The caller owns the storage that the returned string occupies and
+ * is responsible for releasing it.
+ */
+char *fsmonitor__resolve_alias(const char *path,
+ const struct alias_info *info);
+
#endif
--
gitgitgadget
^ permalink raw reply related [flat|nested] 89+ messages in thread* [PATCH v2 05/12] fsmonitor: check for compatability before communicating with fsmonitor
2022-10-14 21:45 ` [PATCH v2 " Eric DeCosta via GitGitGadget
` (3 preceding siblings ...)
2022-10-14 21:45 ` [PATCH v2 04/12] fsmonitor: deal with synthetic firmlinks on macOS Eric DeCosta via GitGitGadget
@ 2022-10-14 21:45 ` Eric DeCosta via GitGitGadget
2022-10-14 21:45 ` [PATCH v2 06/12] fsmonitor: add documentation for allowRemote and socketDir options Eric DeCosta via GitGitGadget
` (9 subsequent siblings)
14 siblings, 0 replies; 89+ messages in thread
From: Eric DeCosta via GitGitGadget @ 2022-10-14 21:45 UTC (permalink / raw)
To: git
Cc: Eric Sunshine, Ævar Arnfjörð Bjarmason,
Eric DeCosta, Eric DeCosta
From: Eric DeCosta <edecosta@mathworks.com>
If fsmonitor is not in a compatible state, warn with an appropriate message.
Signed-off-by: Eric DeCosta <edecosta@mathworks.com>
---
fsmonitor-settings.c | 10 +++++++---
fsmonitor-settings.h | 2 +-
fsmonitor.c | 7 +++++++
3 files changed, 15 insertions(+), 4 deletions(-)
diff --git a/fsmonitor-settings.c b/fsmonitor-settings.c
index 531a1b6f956..ee63a97dc51 100644
--- a/fsmonitor-settings.c
+++ b/fsmonitor-settings.c
@@ -1,6 +1,7 @@
#include "cache.h"
#include "config.h"
#include "repository.h"
+#include "fsmonitor-ipc.h"
#include "fsmonitor-settings.h"
#include "fsmonitor-path-utils.h"
@@ -242,10 +243,11 @@ enum fsmonitor_reason fsm_settings__get_reason(struct repository *r)
return r->settings.fsmonitor->reason;
}
-char *fsm_settings__get_incompatible_msg(const struct repository *r,
+char *fsm_settings__get_incompatible_msg(struct repository *r,
enum fsmonitor_reason reason)
{
struct strbuf msg = STRBUF_INIT;
+ const char *socket_dir;
switch (reason) {
case FSMONITOR_REASON_UNTESTED:
@@ -281,9 +283,11 @@ char *fsm_settings__get_incompatible_msg(const struct repository *r,
goto done;
case FSMONITOR_REASON_NOSOCKETS:
+ socket_dir = dirname((char *)fsmonitor_ipc__get_path(r));
strbuf_addf(&msg,
- _("repository '%s' is incompatible with fsmonitor due to lack of Unix sockets"),
- r->worktree);
+ _("socket directory '%s' is incompatible with fsmonitor due"
+ " to lack of Unix sockets support"),
+ socket_dir);
goto done;
}
diff --git a/fsmonitor-settings.h b/fsmonitor-settings.h
index 0721617b95a..ab02e3995ee 100644
--- a/fsmonitor-settings.h
+++ b/fsmonitor-settings.h
@@ -33,7 +33,7 @@ enum fsmonitor_mode fsm_settings__get_mode(struct repository *r);
const char *fsm_settings__get_hook_path(struct repository *r);
enum fsmonitor_reason fsm_settings__get_reason(struct repository *r);
-char *fsm_settings__get_incompatible_msg(const struct repository *r,
+char *fsm_settings__get_incompatible_msg(struct repository *r,
enum fsmonitor_reason reason);
struct fsmonitor_settings;
diff --git a/fsmonitor.c b/fsmonitor.c
index 57d6a483bee..540736b39fd 100644
--- a/fsmonitor.c
+++ b/fsmonitor.c
@@ -295,6 +295,7 @@ static int fsmonitor_force_update_threshold = 100;
void refresh_fsmonitor(struct index_state *istate)
{
+ static int warn_once = 0;
struct strbuf query_result = STRBUF_INIT;
int query_success = 0, hook_version = -1;
size_t bol = 0; /* beginning of line */
@@ -305,6 +306,12 @@ void refresh_fsmonitor(struct index_state *istate)
int is_trivial = 0;
struct repository *r = istate->repo ? istate->repo : the_repository;
enum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r);
+ enum fsmonitor_reason reason = fsm_settings__get_reason(r);
+
+ if (!warn_once && reason > FSMONITOR_REASON_OK) {
+ warn_once = 1;
+ warning("%s", fsm_settings__get_incompatible_msg(r, reason));
+ }
if (fsm_mode <= FSMONITOR_MODE_DISABLED ||
istate->fsmonitor_has_run_once)
--
gitgitgadget
^ permalink raw reply related [flat|nested] 89+ messages in thread* [PATCH v2 06/12] fsmonitor: add documentation for allowRemote and socketDir options
2022-10-14 21:45 ` [PATCH v2 " Eric DeCosta via GitGitGadget
` (4 preceding siblings ...)
2022-10-14 21:45 ` [PATCH v2 05/12] fsmonitor: check for compatability before communicating with fsmonitor Eric DeCosta via GitGitGadget
@ 2022-10-14 21:45 ` Eric DeCosta via GitGitGadget
2022-10-14 21:45 ` [PATCH v2 07/12] fsmonitor: prepare to share code between Mac OS and Linux Eric DeCosta via GitGitGadget
` (8 subsequent siblings)
14 siblings, 0 replies; 89+ messages in thread
From: Eric DeCosta via GitGitGadget @ 2022-10-14 21:45 UTC (permalink / raw)
To: git
Cc: Eric Sunshine, Ævar Arnfjörð Bjarmason,
Eric DeCosta, Eric DeCosta
From: Eric DeCosta <edecosta@mathworks.com>
Add documentation for 'fsmonitor.allowRemote' and 'fsmonitor.socketDir'.
Call-out experimental nature of 'fsmonitor.allowRemote' and limited
filesystem support for 'fsmonitor.socketDir'.
Signed-off-by: Eric DeCosta <edecosta@mathworks.com>
---
Documentation/config.txt | 2 ++
Documentation/config/fsmonitor--daemon.txt | 11 +++++++
Documentation/git-fsmonitor--daemon.txt | 37 ++++++++++++++++++++--
3 files changed, 47 insertions(+), 3 deletions(-)
create mode 100644 Documentation/config/fsmonitor--daemon.txt
diff --git a/Documentation/config.txt b/Documentation/config.txt
index 5b5b9765699..1e205831656 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -423,6 +423,8 @@ include::config/filter.txt[]
include::config/fsck.txt[]
+include::config/fsmonitor--daemon.txt[]
+
include::config/gc.txt[]
include::config/gitcvs.txt[]
diff --git a/Documentation/config/fsmonitor--daemon.txt b/Documentation/config/fsmonitor--daemon.txt
new file mode 100644
index 00000000000..c225c6c9e74
--- /dev/null
+++ b/Documentation/config/fsmonitor--daemon.txt
@@ -0,0 +1,11 @@
+fsmonitor.allowRemote::
+ By default, the fsmonitor daemon refuses to work against network-mounted
+ repositories. Setting `fsmonitor.allowRemote` to `true` overrides this
+ behavior. Only respected when `core.fsmonitor` is set to `true`.
+
+fsmonitor.socketDir::
+ This Mac OS-specific option, if set, specifies the directory in
+ which to create the Unix domain socket used for communication
+ between the fsmonitor daemon and various Git commands. The directory must
+ reside on a native Mac OS filesystem. Only respected when `core.fsmonitor`
+ is set to `true`.
diff --git a/Documentation/git-fsmonitor--daemon.txt b/Documentation/git-fsmonitor--daemon.txt
index cc142fb8612..8238eadb0e1 100644
--- a/Documentation/git-fsmonitor--daemon.txt
+++ b/Documentation/git-fsmonitor--daemon.txt
@@ -3,7 +3,7 @@ git-fsmonitor{litdd}daemon(1)
NAME
----
-git-fsmonitor--daemon - A Built-in File System Monitor
+git-fsmonitor--daemon - A Built-in Filesystem Monitor
SYNOPSIS
--------
@@ -17,7 +17,7 @@ DESCRIPTION
-----------
A daemon to watch the working directory for file and directory
-changes using platform-specific file system notification facilities.
+changes using platform-specific filesystem notification facilities.
This daemon communicates directly with commands like `git status`
using the link:technical/api-simple-ipc.html[simple IPC] interface
@@ -63,13 +63,44 @@ CAVEATS
-------
The fsmonitor daemon does not currently know about submodules and does
-not know to filter out file system events that happen within a
+not know to filter out filesystem events that happen within a
submodule. If fsmonitor daemon is watching a super repo and a file is
modified within the working directory of a submodule, it will report
the change (as happening against the super repo). However, the client
will properly ignore these extra events, so performance may be affected
but it will not cause an incorrect result.
+By default, the fsmonitor daemon refuses to work against network-mounted
+repositories; this may be overridden by setting `fsmonitor.allowRemote` to
+`true`. Note, however, that the fsmonitor daemon is not guaranteed to work
+correctly with all network-mounted repositories and such use is considered
+experimental.
+
+On Mac OS, the inter-process communication (IPC) between various Git
+commands and the fsmonitor daemon is done via a Unix domain socket (UDS) -- a
+special type of file -- which is supported by native Mac OS filesystems,
+but not on network-mounted filesystems, NTFS, or FAT32. Other filesystems
+may or may not have the needed support; the fsmonitor daemon is not guaranteed
+to work with these filesystems and such use is considered experimental.
+
+By default, the socket is created in the `.git` directory, however, if the
+`.git` directory is on a network-mounted filesystem, it will be instead be
+created at `$HOME/.git-fsmonitor-*` unless `$HOME` itself is on a
+network-mounted filesystem in which case you must set the configuration
+variable `fsmonitor.socketDir` to the path of a directory on a Mac OS native
+filesystem in which to create the socket file.
+
+If none of the above directories (`.git`, `$HOME`, or `fsmonitor.socketDir`)
+is on a native Mac OS file filesystem the fsmonitor daemon will report an
+error that will cause the daemon and the currently running command to exit.
+
+CONFIGURATION
+-------------
+
+include::includes/cmd-config-section-all.txt[]
+
+include::config/fsmonitor--daemon.txt[]
+
GIT
---
Part of the linkgit:git[1] suite
--
gitgitgadget
^ permalink raw reply related [flat|nested] 89+ messages in thread* [PATCH v2 07/12] fsmonitor: prepare to share code between Mac OS and Linux
2022-10-14 21:45 ` [PATCH v2 " Eric DeCosta via GitGitGadget
` (5 preceding siblings ...)
2022-10-14 21:45 ` [PATCH v2 06/12] fsmonitor: add documentation for allowRemote and socketDir options Eric DeCosta via GitGitGadget
@ 2022-10-14 21:45 ` Eric DeCosta via GitGitGadget
2022-10-14 23:46 ` Junio C Hamano
2022-10-14 21:45 ` [PATCH v2 08/12] fsmonitor: determine if filesystem is local or remote Eric DeCosta via GitGitGadget
` (7 subsequent siblings)
14 siblings, 1 reply; 89+ messages in thread
From: Eric DeCosta via GitGitGadget @ 2022-10-14 21:45 UTC (permalink / raw)
To: git
Cc: Eric Sunshine, Ævar Arnfjörð Bjarmason,
Eric DeCosta, Eric DeCosta
From: Eric DeCosta <edecosta@mathworks.com>
Linux and Mac OS can share some of the code originally developed for Mac OS.
Minor update to compat/fsmonitor/fsm-ipc-unix.c to make it cross-platform.
Mac OS and Linux can share fsm-ipc-unix.c
Both platforms can also share compat/fsmonitor/fsm-settings-unix.c but we
will leave room for future, platform-specific checks by having the platform-
specific implementations call into fsm-settings-unix.
Signed-off-by: Eric DeCosta <edecosta@mathworks.com>
---
Makefile | 9 ++-
compat/fsmonitor/fsm-health-linux.c | 24 +++++++
.../{fsm-ipc-darwin.c => fsm-ipc-unix.c} | 8 +--
compat/fsmonitor/fsm-settings-darwin.c | 57 +----------------
compat/fsmonitor/fsm-settings-linux.c | 11 ++++
compat/fsmonitor/fsm-settings-unix.c | 62 +++++++++++++++++++
compat/fsmonitor/fsm-settings-unix.h | 11 ++++
config.mak.uname | 1 +
contrib/buildsystems/CMakeLists.txt | 17 ++++-
9 files changed, 138 insertions(+), 62 deletions(-)
create mode 100644 compat/fsmonitor/fsm-health-linux.c
rename compat/fsmonitor/{fsm-ipc-darwin.c => fsm-ipc-unix.c} (89%)
create mode 100644 compat/fsmonitor/fsm-settings-linux.c
create mode 100644 compat/fsmonitor/fsm-settings-unix.c
create mode 100644 compat/fsmonitor/fsm-settings-unix.h
diff --git a/Makefile b/Makefile
index 9553d590824..bbbde00ce49 100644
--- a/Makefile
+++ b/Makefile
@@ -2036,13 +2036,20 @@ endif
ifdef FSMONITOR_DAEMON_BACKEND
COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND
+ ifdef FSMONITOR_DAEMON_COMMON
+ COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_DAEMON_COMMON).o
+ else
+ COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_DAEMON_BACKEND).o
+ endif
COMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o
COMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_BACKEND).o
- COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_DAEMON_BACKEND).o
endif
ifdef FSMONITOR_OS_SETTINGS
COMPAT_CFLAGS += -DHAVE_FSMONITOR_OS_SETTINGS
+ ifdef FSMONITOR_DAEMON_COMMON
+ COMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_DAEMON_COMMON).o
+ endif
COMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_OS_SETTINGS).o
COMPAT_OBJS += compat/fsmonitor/fsm-path-utils-$(FSMONITOR_OS_SETTINGS).o
endif
diff --git a/compat/fsmonitor/fsm-health-linux.c b/compat/fsmonitor/fsm-health-linux.c
new file mode 100644
index 00000000000..b9f709e8548
--- /dev/null
+++ b/compat/fsmonitor/fsm-health-linux.c
@@ -0,0 +1,24 @@
+#include "cache.h"
+#include "config.h"
+#include "fsmonitor.h"
+#include "fsm-health.h"
+#include "fsmonitor--daemon.h"
+
+int fsm_health__ctor(struct fsmonitor_daemon_state *state)
+{
+ return 0;
+}
+
+void fsm_health__dtor(struct fsmonitor_daemon_state *state)
+{
+ return;
+}
+
+void fsm_health__loop(struct fsmonitor_daemon_state *state)
+{
+ return;
+}
+
+void fsm_health__stop_async(struct fsmonitor_daemon_state *state)
+{
+}
diff --git a/compat/fsmonitor/fsm-ipc-darwin.c b/compat/fsmonitor/fsm-ipc-unix.c
similarity index 89%
rename from compat/fsmonitor/fsm-ipc-darwin.c
rename to compat/fsmonitor/fsm-ipc-unix.c
index ce843d63348..3ba3b9e17ed 100644
--- a/compat/fsmonitor/fsm-ipc-darwin.c
+++ b/compat/fsmonitor/fsm-ipc-unix.c
@@ -10,7 +10,7 @@ static GIT_PATH_FUNC(fsmonitor_ipc__get_default_path, "fsmonitor--daemon.ipc")
const char *fsmonitor_ipc__get_path(struct repository *r)
{
static const char *ipc_path = NULL;
- SHA_CTX sha1ctx;
+ git_SHA_CTX sha1ctx;
char *sock_dir = NULL;
struct strbuf ipc_file = STRBUF_INIT;
unsigned char hash[SHA_DIGEST_LENGTH];
@@ -28,9 +28,9 @@ const char *fsmonitor_ipc__get_path(struct repository *r)
return ipc_path;
}
- SHA1_Init(&sha1ctx);
- SHA1_Update(&sha1ctx, r->worktree, strlen(r->worktree));
- SHA1_Final(hash, &sha1ctx);
+ git_SHA1_Init(&sha1ctx);
+ git_SHA1_Update(&sha1ctx, r->worktree, strlen(r->worktree));
+ git_SHA1_Final(hash, &sha1ctx);
repo_config_get_string(r, "fsmonitor.socketdir", &sock_dir);
diff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c
index 6abbc7af3ab..2f5c6a88bf0 100644
--- a/compat/fsmonitor/fsm-settings-darwin.c
+++ b/compat/fsmonitor/fsm-settings-darwin.c
@@ -1,62 +1,11 @@
#include "config.h"
#include "fsmonitor.h"
#include "fsmonitor-ipc.h"
-#include "fsmonitor-settings.h"
#include "fsmonitor-path-utils.h"
-
- /*
- * For the builtin FSMonitor, we create the Unix domain socket for the
- * IPC in the .git directory. If the working directory is remote,
- * then the socket will be created on the remote file system. This
- * can fail if the remote file system does not support UDS file types
- * (e.g. smbfs to a Windows server) or if the remote kernel does not
- * allow a non-local process to bind() the socket. (These problems
- * could be fixed by moving the UDS out of the .git directory and to a
- * well-known local directory on the client machine, but care should
- * be taken to ensure that $HOME is actually local and not a managed
- * file share.)
- *
- * FAT32 and NTFS working directories are problematic too.
- *
- * The builtin FSMonitor uses a Unix domain socket in the .git
- * directory for IPC. These Windows drive formats do not support
- * Unix domain sockets, so mark them as incompatible for the daemon.
- *
- */
-static enum fsmonitor_reason check_uds_volume(struct repository *r)
-{
- struct fs_info fs;
- const char *ipc_path = fsmonitor_ipc__get_path(r);
- struct strbuf path = STRBUF_INIT;
- strbuf_add(&path, ipc_path, strlen(ipc_path));
-
- if (fsmonitor__get_fs_info(dirname(path.buf), &fs) == -1) {
- strbuf_release(&path);
- return FSMONITOR_REASON_ERROR;
- }
-
- strbuf_release(&path);
-
- if (fs.is_remote ||
- !strcmp(fs.typename, "msdos") ||
- !strcmp(fs.typename, "ntfs")) {
- free(fs.typename);
- return FSMONITOR_REASON_NOSOCKETS;
- }
-
- free(fs.typename);
- return FSMONITOR_REASON_OK;
-}
+#include "fsmonitor-settings.h"
+#include "fsm-settings-unix.h"
enum fsmonitor_reason fsm_os__incompatible(struct repository *r, int ipc)
{
- enum fsmonitor_reason reason;
-
- if (ipc) {
- reason = check_uds_volume(r);
- if (reason != FSMONITOR_REASON_OK)
- return reason;
- }
-
- return FSMONITOR_REASON_OK;
+ return fsm_os__incompatible_unix(r, ipc);
}
diff --git a/compat/fsmonitor/fsm-settings-linux.c b/compat/fsmonitor/fsm-settings-linux.c
new file mode 100644
index 00000000000..2f5c6a88bf0
--- /dev/null
+++ b/compat/fsmonitor/fsm-settings-linux.c
@@ -0,0 +1,11 @@
+#include "config.h"
+#include "fsmonitor.h"
+#include "fsmonitor-ipc.h"
+#include "fsmonitor-path-utils.h"
+#include "fsmonitor-settings.h"
+#include "fsm-settings-unix.h"
+
+enum fsmonitor_reason fsm_os__incompatible(struct repository *r, int ipc)
+{
+ return fsm_os__incompatible_unix(r, ipc);
+}
diff --git a/compat/fsmonitor/fsm-settings-unix.c b/compat/fsmonitor/fsm-settings-unix.c
new file mode 100644
index 00000000000..6b43e26720e
--- /dev/null
+++ b/compat/fsmonitor/fsm-settings-unix.c
@@ -0,0 +1,62 @@
+#include "config.h"
+#include "fsmonitor.h"
+#include "fsmonitor-ipc.h"
+#include "fsmonitor-path-utils.h"
+#include "fsm-settings-unix.h"
+
+ /*
+ * For the builtin FSMonitor, we create the Unix domain socket for the
+ * IPC in the .git directory. If the working directory is remote,
+ * then the socket will be created on the remote file system. This
+ * can fail if the remote file system does not support UDS file types
+ * (e.g. smbfs to a Windows server) or if the remote kernel does not
+ * allow a non-local process to bind() the socket. (These problems
+ * could be fixed by moving the UDS out of the .git directory and to a
+ * well-known local directory on the client machine, but care should
+ * be taken to ensure that $HOME is actually local and not a managed
+ * file share.)
+ *
+ * FAT32 and NTFS working directories are problematic too.
+ *
+ * The builtin FSMonitor uses a Unix domain socket in the .git
+ * directory for IPC. These Windows drive formats do not support
+ * Unix domain sockets, so mark them as incompatible for the daemon.
+ *
+ */
+static enum fsmonitor_reason check_uds_volume(struct repository *r)
+{
+ struct fs_info fs;
+ const char *ipc_path = fsmonitor_ipc__get_path(r);
+ struct strbuf path = STRBUF_INIT;
+ strbuf_add(&path, ipc_path, strlen(ipc_path));
+
+ if (fsmonitor__get_fs_info(dirname(path.buf), &fs) == -1) {
+ strbuf_release(&path);
+ return FSMONITOR_REASON_ERROR;
+ }
+
+ strbuf_release(&path);
+
+ if (fs.is_remote ||
+ !strcmp(fs.typename, "msdos") ||
+ !strcmp(fs.typename, "ntfs")) {
+ free(fs.typename);
+ return FSMONITOR_REASON_NOSOCKETS;
+ }
+
+ free(fs.typename);
+ return FSMONITOR_REASON_OK;
+}
+
+enum fsmonitor_reason fsm_os__incompatible_unix(struct repository *r, int ipc)
+{
+ enum fsmonitor_reason reason;
+
+ if (ipc) {
+ reason = check_uds_volume(r);
+ if (reason != FSMONITOR_REASON_OK)
+ return reason;
+ }
+
+ return FSMONITOR_REASON_OK;
+}
diff --git a/compat/fsmonitor/fsm-settings-unix.h b/compat/fsmonitor/fsm-settings-unix.h
new file mode 100644
index 00000000000..8a62971190b
--- /dev/null
+++ b/compat/fsmonitor/fsm-settings-unix.h
@@ -0,0 +1,11 @@
+#ifndef FSM_SETTINGS_UNIX_H
+#define FSM_SETTINGS_UNIX_H
+
+#ifdef HAVE_FSMONITOR_OS_SETTINGS
+/*
+ * Check for compatibility on unix-like systems (e.g. Darwin and Linux)
+ */
+enum fsmonitor_reason fsm_os__incompatible_unix(struct repository *r, int ipc);
+#endif /* HAVE_FSMONITOR_OS_SETTINGS */
+
+#endif /* FSM_SETTINGS_UNIX_H */
diff --git a/config.mak.uname b/config.mak.uname
index d63629fe807..9f716cfba81 100644
--- a/config.mak.uname
+++ b/config.mak.uname
@@ -165,6 +165,7 @@ ifeq ($(uname_S),Darwin)
ifndef NO_UNIX_SOCKETS
FSMONITOR_DAEMON_BACKEND = darwin
FSMONITOR_OS_SETTINGS = darwin
+ FSMONITOR_DAEMON_COMMON = unix
endif
endif
diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt
index 787738e6fa3..4c6e84da346 100644
--- a/contrib/buildsystems/CMakeLists.txt
+++ b/contrib/buildsystems/CMakeLists.txt
@@ -304,7 +304,17 @@ else()
endif()
if(SUPPORTS_SIMPLE_IPC)
- if(CMAKE_SYSTEM_NAME STREQUAL "Windows")
+ if(CMAKE_SYSTEM_NAME STREQUAL "Linux")
+ add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)
+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-unix.c)
+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-linux.c)
+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-linux.c)
+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-linux.c)
+
+ add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)
+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-unix.c)
+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-linux.c)
+ elseif(CMAKE_SYSTEM_NAME STREQUAL "Windows")
add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)
list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)
list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-win32.c)
@@ -315,12 +325,13 @@ if(SUPPORTS_SIMPLE_IPC)
list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c)
elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin")
add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)
- list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)
+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-unix.c)
list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-darwin.c)
- list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-darwin.c)
+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)
list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-darwin.c)
add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)
+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-unix.c)
list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c)
endif()
endif()
--
gitgitgadget
^ permalink raw reply related [flat|nested] 89+ messages in thread* Re: [PATCH v2 07/12] fsmonitor: prepare to share code between Mac OS and Linux
2022-10-14 21:45 ` [PATCH v2 07/12] fsmonitor: prepare to share code between Mac OS and Linux Eric DeCosta via GitGitGadget
@ 2022-10-14 23:46 ` Junio C Hamano
2022-10-17 21:30 ` Eric DeCosta
0 siblings, 1 reply; 89+ messages in thread
From: Junio C Hamano @ 2022-10-14 23:46 UTC (permalink / raw)
To: Eric DeCosta via GitGitGadget
Cc: git, Eric Sunshine, Ævar Arnfjörð Bjarmason,
Eric DeCosta
"Eric DeCosta via GitGitGadget" <gitgitgadget@gmail.com> writes:
> ifdef FSMONITOR_DAEMON_BACKEND
> COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND
> + ifdef FSMONITOR_DAEMON_COMMON
> + COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_DAEMON_COMMON).o
> + else
> + COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_DAEMON_BACKEND).o
> + endif
> COMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o
> COMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_BACKEND).o
> - COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_DAEMON_BACKEND).o
> endif
>
> ifdef FSMONITOR_OS_SETTINGS
> COMPAT_CFLAGS += -DHAVE_FSMONITOR_OS_SETTINGS
> + ifdef FSMONITOR_DAEMON_COMMON
> + COMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_DAEMON_COMMON).o
> + endif
> COMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_OS_SETTINGS).o
> COMPAT_OBJS += compat/fsmonitor/fsm-path-utils-$(FSMONITOR_OS_SETTINGS).o
> endif
Ugly.
One overrides backend with common, while the other one doesn't.
That asymmetry alone should stop us and wonder if there is something
fishy in the approach that can be improved. It makes it look like
the word "common" means something quite different between the -ipc
and the -settings world.
I suspect that in both, you should not expose "unix" to this part of
the Makefile. Linux and macOS occasionally being similar in some
places does not have to be exposed here. INstead you can use
backend "linux" and "macos", whose C sources may include from a
separate C source file whose name may contain "unix". That would
allow you to get rid of FSMONITOR_DAEMON_COMMON in a cleaner way.
^ permalink raw reply [flat|nested] 89+ messages in thread
* RE: [PATCH v2 07/12] fsmonitor: prepare to share code between Mac OS and Linux
2022-10-14 23:46 ` Junio C Hamano
@ 2022-10-17 21:30 ` Eric DeCosta
2022-10-18 6:31 ` Junio C Hamano
0 siblings, 1 reply; 89+ messages in thread
From: Eric DeCosta @ 2022-10-17 21:30 UTC (permalink / raw)
To: Junio C Hamano, Eric DeCosta via GitGitGadget
Cc: git@vger.kernel.org, Eric Sunshine,
Ævar Arnfjörð Bjarmason
> -----Original Message-----
> From: Junio C Hamano <gitster@pobox.com>
> Sent: Friday, October 14, 2022 7:46 PM
> To: Eric DeCosta via GitGitGadget <gitgitgadget@gmail.com>
> Cc: git@vger.kernel.org; Eric Sunshine <sunshine@sunshineco.com>; Ævar
> Arnfjörð Bjarmason <avarab@gmail.com>; Eric DeCosta
> <edecosta@mathworks.com>
> Subject: Re: [PATCH v2 07/12] fsmonitor: prepare to share code between
> Mac OS and Linux
>
> "Eric DeCosta via GitGitGadget" <gitgitgadget@gmail.com> writes:
>
> > ifdef FSMONITOR_DAEMON_BACKEND
> > COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND
> > + ifdef FSMONITOR_DAEMON_COMMON
> > + COMPAT_OBJS += compat/fsmonitor/fsm-ipc-
> $(FSMONITOR_DAEMON_COMMON).o
> > + else
> > + COMPAT_OBJS += compat/fsmonitor/fsm-ipc-
> $(FSMONITOR_DAEMON_BACKEND).o
> > + endif
> > COMPAT_OBJS += compat/fsmonitor/fsm-listen-
> $(FSMONITOR_DAEMON_BACKEND).o
> > COMPAT_OBJS += compat/fsmonitor/fsm-health-
> $(FSMONITOR_DAEMON_BACKEND).o
> > - COMPAT_OBJS += compat/fsmonitor/fsm-ipc-
> $(FSMONITOR_DAEMON_BACKEND).o
> > endif
> >
> > ifdef FSMONITOR_OS_SETTINGS
> > COMPAT_CFLAGS += -DHAVE_FSMONITOR_OS_SETTINGS
> > + ifdef FSMONITOR_DAEMON_COMMON
> > + COMPAT_OBJS += compat/fsmonitor/fsm-settings-
> $(FSMONITOR_DAEMON_COMMON).o
> > + endif
> > COMPAT_OBJS += compat/fsmonitor/fsm-settings-
> $(FSMONITOR_OS_SETTINGS).o
> > COMPAT_OBJS +=
> > compat/fsmonitor/fsm-path-utils-$(FSMONITOR_OS_SETTINGS).o
> > endif
>
> Ugly.
>
> One overrides backend with common, while the other one doesn't.
> That asymmetry alone should stop us and wonder if there is something fishy
> in the approach that can be improved. It makes it look like the word
> "common" means something quite different between the -ipc and the -
> settings world.
>
> I suspect that in both, you should not expose "unix" to this part of the
> Makefile. Linux and macOS occasionally being similar in some places does
> not have to be exposed here. INstead you can use backend "linux" and
> "macos", whose C sources may include from a separate C source file whose
> name may contain "unix". That would allow you to get rid of
> FSMONITOR_DAEMON_COMMON in a cleaner way.
Let me see if I am understanding you correctly. Are you suggesting something like:
ifdef FSMONITOR_DAEMON_BACKEND_LINUX
COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND
COMPAT_OBJS += compat/fsmonitor/fsm-ipc-unix.o
COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_DAEMON_BACKEND_LINUX).o
COMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND_LINUX).o
COMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_BACKEND_LINUX).o
endif
ifdef FSMONITOR_DAEMON_BACKEND_DARWIN
COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND
COMPAT_OBJS += compat/fsmonitor/fsm-ipc-unix.o
COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_DAEMON_BACKEND_DARWIN).o
COMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND_DARWIN).o
COMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_BACKEND_DARWIN).o
endif
And similarly for settings:
ifdef FSMONITOR_DAEMON_SETTINGS_LINUX
COMPAT_CFLAGS += -DHAVE_FSMONITOR_OS_SETTINGS
COMPAT_OBJS += compat/fsmonitor/fsm-settings-unix.o
COMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_DAEMON_SETTINGS_LINUX).o
COMPAT_OBJS += compat/fsmonitor/fsm-path-utils-$(FSMONITOR_DAEMON_SETTINGS_LINUX).o
endif
ifdef FSMONITOR_DAEMON_SETTINGS_DARWIN
COMPAT_CFLAGS += -DHAVE_FSMONITOR_OS_SETTINGS
COMPAT_OBJS += compat/fsmonitor/fsm-settings-unix.o
COMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_DAEMON_SETTINGS_DARWIN).o
COMPAT_OBJS += compat/fsmonitor/fsm-path-utils-$(FSMONITOR_DAEMON_SETTINGS_DARWIN).o
endif
And the rest of the platforms remain as was, with FSMONITOR_DAEMON_BACKEND and FSMONITOR_OS_SETTINGS
-Eric
^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [PATCH v2 07/12] fsmonitor: prepare to share code between Mac OS and Linux
2022-10-17 21:30 ` Eric DeCosta
@ 2022-10-18 6:31 ` Junio C Hamano
0 siblings, 0 replies; 89+ messages in thread
From: Junio C Hamano @ 2022-10-18 6:31 UTC (permalink / raw)
To: Eric DeCosta
Cc: Eric DeCosta via GitGitGadget, git@vger.kernel.org, Eric Sunshine,
Ævar Arnfjörð Bjarmason
Eric DeCosta <edecosta@mathworks.com> writes:
>> Ugly.
>>
>> One overrides backend with common, while the other one doesn't.
>> That asymmetry alone should stop us and wonder if there is something fishy
>> in the approach that can be improved. It makes it look like the word
>> "common" means something quite different between the -ipc and the -
>> settings world.
>>
>> I suspect that in both, you should not expose "unix" to this part of the
>> Makefile. Linux and macOS occasionally being similar in some places does
>> not have to be exposed here. INstead you can use backend "linux" and
>> "macos", whose C sources may include from a separate C source file whose
>> name may contain "unix". That would allow you to get rid of
>> FSMONITOR_DAEMON_COMMON in a cleaner way.
>
> Let me see if I am understanding you correctly. Are you suggesting something like:
Not really.
What I had in mind was an arrangement more like
$ for variant in linux macos; do grep include fsm-ipc-$variant.c; done
#include "fsm-ipc-common-unix.cinclude"
#include "fsm-ipc-common-unix.cinclude"
and COMPAT_OBJS knowning only about fsm-ipc-linux.o or
fsm-ipc-macos.o, depending on which platform you are building.
^ permalink raw reply [flat|nested] 89+ messages in thread
* [PATCH v2 08/12] fsmonitor: determine if filesystem is local or remote
2022-10-14 21:45 ` [PATCH v2 " Eric DeCosta via GitGitGadget
` (6 preceding siblings ...)
2022-10-14 21:45 ` [PATCH v2 07/12] fsmonitor: prepare to share code between Mac OS and Linux Eric DeCosta via GitGitGadget
@ 2022-10-14 21:45 ` Eric DeCosta via GitGitGadget
2022-10-14 21:45 ` [PATCH v2 09/12] fsmonitor: implement filesystem change listener for Linux Eric DeCosta via GitGitGadget
` (6 subsequent siblings)
14 siblings, 0 replies; 89+ messages in thread
From: Eric DeCosta via GitGitGadget @ 2022-10-14 21:45 UTC (permalink / raw)
To: git
Cc: Eric Sunshine, Ævar Arnfjörð Bjarmason,
Eric DeCosta, Eric DeCosta
From: Eric DeCosta <edecosta@mathworks.com>
Compare the given path to the mounted filesystems. Find the mount that is
the longest prefix of the path (if any) and determine if that mount is on a
local or remote filesystem.
Signed-off-by: Eric DeCosta <edecosta@mathworks.com>
---
compat/fsmonitor/fsm-path-utils-linux.c | 169 ++++++++++++++++++++++++
1 file changed, 169 insertions(+)
create mode 100644 compat/fsmonitor/fsm-path-utils-linux.c
diff --git a/compat/fsmonitor/fsm-path-utils-linux.c b/compat/fsmonitor/fsm-path-utils-linux.c
new file mode 100644
index 00000000000..039f044b670
--- /dev/null
+++ b/compat/fsmonitor/fsm-path-utils-linux.c
@@ -0,0 +1,169 @@
+#include "fsmonitor.h"
+#include "fsmonitor-path-utils.h"
+#include <errno.h>
+#include <mntent.h>
+#include <sys/mount.h>
+#include <sys/statvfs.h>
+
+/*
+ * https://github.com/coreutils/gnulib/blob/master/lib/mountlist.c
+ */
+#ifndef ME_REMOTE
+/* A file system is "remote" if its Fs_name contains a ':'
+ or if (it is of type (smbfs or cifs) and its Fs_name starts with '//')
+ or if it is of any other of the listed types
+ or Fs_name is equal to "-hosts" (used by autofs to mount remote fs).
+ "VM" file systems like prl_fs or vboxsf are not considered remote here. */
+# define ME_REMOTE(Fs_name, Fs_type) \
+ (strchr (Fs_name, ':') != NULL \
+ || ((Fs_name)[0] == '/' \
+ && (Fs_name)[1] == '/' \
+ && (strcmp (Fs_type, "smbfs") == 0 \
+ || strcmp (Fs_type, "smb3") == 0 \
+ || strcmp (Fs_type, "cifs") == 0)) \
+ || strcmp (Fs_type, "acfs") == 0 \
+ || strcmp (Fs_type, "afs") == 0 \
+ || strcmp (Fs_type, "coda") == 0 \
+ || strcmp (Fs_type, "auristorfs") == 0 \
+ || strcmp (Fs_type, "fhgfs") == 0 \
+ || strcmp (Fs_type, "gpfs") == 0 \
+ || strcmp (Fs_type, "ibrix") == 0 \
+ || strcmp (Fs_type, "ocfs2") == 0 \
+ || strcmp (Fs_type, "vxfs") == 0 \
+ || strcmp ("-hosts", Fs_name) == 0)
+#endif
+
+static int find_mount(const char *path, const struct statvfs *fs,
+ struct mntent *ent)
+{
+ const char *const mounts = "/proc/mounts";
+ const char *rp = real_pathdup(path, 1);
+ struct mntent *ment = NULL;
+ struct statvfs mntfs;
+ FILE *fp;
+ int found = 0;
+ int dlen, plen, flen = 0;
+
+ ent->mnt_fsname = NULL;
+ ent->mnt_dir = NULL;
+ ent->mnt_type = NULL;
+
+ fp = setmntent(mounts, "r");
+ if (!fp) {
+ error_errno(_("setmntent('%s') failed"), mounts);
+ return -1;
+ }
+
+ plen = strlen(rp);
+
+ /* read all the mount information and compare to path */
+ while ((ment = getmntent(fp)) != NULL) {
+ if (statvfs(ment->mnt_dir, &mntfs)) {
+ switch (errno) {
+ case EPERM:
+ case ESRCH:
+ case EACCES:
+ continue;
+ default:
+ error_errno(_("statvfs('%s') failed"), ment->mnt_dir);
+ endmntent(fp);
+ return -1;
+ }
+ }
+
+ /* is mount on the same filesystem and is a prefix of the path */
+ if ((fs->f_fsid == mntfs.f_fsid) &&
+ !strncmp(ment->mnt_dir, rp, strlen(ment->mnt_dir))) {
+ dlen = strlen(ment->mnt_dir);
+ if (dlen > plen)
+ continue;
+ /*
+ * root is always a potential match; otherwise look for
+ * directory prefix
+ */
+ if ((dlen == 1 && ment->mnt_dir[0] == '/') ||
+ (dlen > flen && (!rp[dlen] || rp[dlen] == '/'))) {
+ flen = dlen;
+ /*
+ * https://man7.org/linux/man-pages/man3/getmntent.3.html
+ *
+ * The pointer points to a static area of memory which is
+ * overwritten by subsequent calls to getmntent().
+ */
+ found = 1;
+ free(ent->mnt_fsname);
+ free(ent->mnt_dir);
+ free(ent->mnt_type);
+ ent->mnt_fsname = xstrdup(ment->mnt_fsname);
+ ent->mnt_dir = xstrdup(ment->mnt_dir);
+ ent->mnt_type = xstrdup(ment->mnt_type);
+ }
+ }
+ }
+ endmntent(fp);
+
+ if (!found)
+ return -1;
+
+ return 0;
+}
+
+int fsmonitor__get_fs_info(const char *path, struct fs_info *fs_info)
+{
+ struct mntent ment;
+ struct statvfs fs;
+
+ if (statvfs(path, &fs))
+ return error_errno(_("statvfs('%s') failed"), path);
+
+
+ if (find_mount(path, &fs, &ment) < 0) {
+ free(ment.mnt_fsname);
+ free(ment.mnt_dir);
+ free(ment.mnt_type);
+ return -1;
+ }
+
+ trace_printf_key(&trace_fsmonitor,
+ "statvfs('%s') [flags 0x%08lx] '%s' '%s'",
+ path, fs.f_flag, ment.mnt_type, ment.mnt_fsname);
+
+ fs_info->is_remote = ME_REMOTE(ment.mnt_fsname, ment.mnt_type);
+ fs_info->typename = ment.mnt_fsname;
+ free(ment.mnt_dir);
+ free(ment.mnt_type);
+
+ trace_printf_key(&trace_fsmonitor,
+ "'%s' is_remote: %d",
+ path, fs_info->is_remote);
+ return 0;
+}
+
+int fsmonitor__is_fs_remote(const char *path)
+{
+ struct fs_info fs;
+
+ if (fsmonitor__get_fs_info(path, &fs))
+ return -1;
+
+ free(fs.typename);
+
+ return fs.is_remote;
+}
+
+/*
+ * No-op for now.
+ */
+int fsmonitor__get_alias(const char *path, struct alias_info *info)
+{
+ return 0;
+}
+
+/*
+ * No-op for now.
+ */
+char *fsmonitor__resolve_alias(const char *path,
+ const struct alias_info *info)
+{
+ return NULL;
+}
--
gitgitgadget
^ permalink raw reply related [flat|nested] 89+ messages in thread* [PATCH v2 09/12] fsmonitor: implement filesystem change listener for Linux
2022-10-14 21:45 ` [PATCH v2 " Eric DeCosta via GitGitGadget
` (7 preceding siblings ...)
2022-10-14 21:45 ` [PATCH v2 08/12] fsmonitor: determine if filesystem is local or remote Eric DeCosta via GitGitGadget
@ 2022-10-14 21:45 ` Eric DeCosta via GitGitGadget
2022-10-18 11:59 ` Ævar Arnfjörð Bjarmason
2022-10-14 21:45 ` [PATCH v2 10/12] fsmonitor: enable fsmonitor " Eric DeCosta via GitGitGadget
` (5 subsequent siblings)
14 siblings, 1 reply; 89+ messages in thread
From: Eric DeCosta via GitGitGadget @ 2022-10-14 21:45 UTC (permalink / raw)
To: git
Cc: Eric Sunshine, Ævar Arnfjörð Bjarmason,
Eric DeCosta, Eric DeCosta
From: Eric DeCosta <edecosta@mathworks.com>
Implement a filesystem change listener for Linux based on the inotify API:
https://man7.org/linux/man-pages/man7/inotify.7.html
inotify requires registering a watch on every directory in the worktree and
special handling of moves/renames.
Signed-off-by: Eric DeCosta <edecosta@mathworks.com>
---
compat/fsmonitor/fsm-listen-linux.c | 664 ++++++++++++++++++++++++++++
1 file changed, 664 insertions(+)
create mode 100644 compat/fsmonitor/fsm-listen-linux.c
diff --git a/compat/fsmonitor/fsm-listen-linux.c b/compat/fsmonitor/fsm-listen-linux.c
new file mode 100644
index 00000000000..0ac131a9497
--- /dev/null
+++ b/compat/fsmonitor/fsm-listen-linux.c
@@ -0,0 +1,664 @@
+#include "cache.h"
+#include "fsmonitor.h"
+#include "fsm-listen.h"
+#include "fsmonitor--daemon.h"
+#include <dirent.h>
+#include <fcntl.h>
+#include <sys/inotify.h>
+#include <sys/stat.h>
+
+/*
+ * Safe value to bitwise OR with rest of mask for
+ * kernels that do not support IN_MASK_CREATE
+ */
+#ifndef IN_MASK_CREATE
+#define IN_MASK_CREATE 0x00000000
+#endif
+
+enum shutdown_reason {
+ SHUTDOWN_CONTINUE = 0,
+ SHUTDOWN_STOP,
+ SHUTDOWN_ERROR,
+ SHUTDOWN_FORCE
+};
+
+struct watch_entry {
+ struct hashmap_entry ent;
+ int wd;
+ uint32_t cookie;
+ const char *dir;
+};
+
+struct rename_entry {
+ struct hashmap_entry ent;
+ time_t whence;
+ uint32_t cookie;
+ const char *dir;
+};
+
+struct fsm_listen_data {
+ int fd_inotify;
+ enum shutdown_reason shutdown;
+ struct hashmap watches;
+ struct hashmap renames;
+ struct hashmap revwatches;
+};
+
+static int watch_entry_cmp(const void *cmp_data,
+ const struct hashmap_entry *eptr,
+ const struct hashmap_entry *entry_or_key,
+ const void *keydata)
+{
+ const struct watch_entry *e1, *e2;
+
+ e1 = container_of(eptr, const struct watch_entry, ent);
+ e2 = container_of(eptr, const struct watch_entry, ent);
+ return e1->wd != e2->wd;
+}
+
+static int revwatches_entry_cmp(const void *cmp_data,
+ const struct hashmap_entry *eptr,
+ const struct hashmap_entry *entry_or_key,
+ const void *keydata)
+{
+ const struct watch_entry *e1, *e2;
+
+ e1 = container_of(eptr, const struct watch_entry, ent);
+ e2 = container_of(eptr, const struct watch_entry, ent);
+ return strcmp(e1->dir, e2->dir);
+}
+
+static int rename_entry_cmp(const void *cmp_data,
+ const struct hashmap_entry *eptr,
+ const struct hashmap_entry *entry_or_key,
+ const void *keydata)
+{
+ const struct rename_entry *e1, *e2;
+
+ e1 = container_of(eptr, const struct rename_entry, ent);
+ e2 = container_of(eptr, const struct rename_entry, ent);
+ return e1->cookie != e2->cookie;
+}
+
+/*
+ * Register an inotify watch, add watch descriptor to path mapping
+ * and the reverse mapping.
+ */
+static int add_watch(const char *path, struct fsm_listen_data *data)
+{
+ const char *interned = strintern(path);
+ struct watch_entry *w1, *w2;
+
+ /* add the inotify watch, don't allow watches to be modified */
+ int wd = inotify_add_watch(data->fd_inotify, interned,
+ (IN_ALL_EVENTS | IN_ONLYDIR | IN_MASK_CREATE)
+ ^ IN_ACCESS ^ IN_CLOSE ^ IN_OPEN);
+ if (wd < 0)
+ return error_errno("inotify_add_watch('%s') failed", interned);
+
+ /* add watch descriptor -> directory mapping */
+ CALLOC_ARRAY(w1, 1);
+ w1->wd = wd;
+ w1->dir = interned;
+ hashmap_entry_init(&w1->ent, memhash(&w1->wd, sizeof(int)));
+ hashmap_add(&data->watches, &w1->ent);
+
+ /* add directory -> watch descriptor mapping */
+ CALLOC_ARRAY(w2, 1);
+ w2->wd = wd;
+ w2->dir = interned;
+ hashmap_entry_init(&w2->ent, memhash(w2->dir, strlen(w2->dir)));
+ hashmap_add(&data->revwatches, &w2->ent);
+
+ return 0;
+}
+
+/*
+ * Remove the inotify watch, the watch descriptor to path mapping
+ * and the reverse mapping.
+ */
+static void remove_watch(struct watch_entry *w,
+ struct fsm_listen_data *data)
+{
+ struct watch_entry k1, k2, *w1, *w2;
+
+ /* remove watch, ignore error if kernel already did it */
+ if (inotify_rm_watch(data->fd_inotify, w->wd) && errno != EINVAL)
+ error_errno("inotify_rm_watch() failed");
+
+ hashmap_entry_init(&k1.ent, memhash(&w->wd, sizeof(int)));
+ w1 = hashmap_remove_entry(&data->watches, &k1, ent, NULL);
+ if (!w1)
+ BUG("Double remove of watch for '%s'", w->dir);
+
+ if (w1->cookie)
+ BUG("Removing watch for '%s' which has a pending rename", w1->dir);
+
+ hashmap_entry_init(&k2.ent, memhash(w->dir, strlen(w->dir)));
+ w2 = hashmap_remove_entry(&data->revwatches, &k2, ent, NULL);
+ if (!w2)
+ BUG("Double remove of reverse watch for '%s'", w->dir);
+
+ /* w1->dir and w2->dir are interned strings, we don't own them */
+ free(w1);
+ free(w2);
+}
+
+/*
+ * Check for stale directory renames.
+ *
+ * https://man7.org/linux/man-pages/man7/inotify.7.html
+ *
+ * Allow for some small timeout to account for the fact that insertion of the
+ * IN_MOVED_FROM+IN_MOVED_TO event pair is not atomic, and the possibility that
+ * there may not be any IN_MOVED_TO event.
+ *
+ * If the IN_MOVED_TO event is not received within the timeout then events have
+ * been missed and the monitor is in an inconsistent state with respect to the
+ * filesystem.
+ */
+static int check_stale_dir_renames(struct hashmap *renames, time_t max_age)
+{
+ struct rename_entry *re;
+ struct hashmap_iter iter;
+
+ hashmap_for_each_entry(renames, &iter, re, ent) {
+ if (re->whence <= max_age)
+ return -1;
+ }
+ return 0;
+}
+
+/*
+ * Track pending renames.
+ *
+ * Tracking is done via a event cookie to watch descriptor mapping.
+ *
+ * A rename is not complete until matching a IN_MOVED_TO event is received
+ * for a corresponding IN_MOVED_FROM event.
+ */
+static void add_dir_rename(uint32_t cookie, const char *path,
+ struct fsm_listen_data *data)
+{
+ struct watch_entry k, *w;
+ struct rename_entry *re;
+
+ /* lookup the watch descriptor for the given path */
+ hashmap_entry_init(&k.ent, memhash(path, strlen(path)));
+ w = hashmap_get_entry(&data->revwatches, &k, ent, NULL);
+ if (!w) /* should never happen */
+ BUG("No watch for '%s'", path);
+ w->cookie = cookie;
+
+ /* add the pending rename to match against later */
+ CALLOC_ARRAY(re, 1);
+ re->dir = w->dir;
+ re->cookie = w->cookie;
+ re->whence = time(NULL);
+ hashmap_entry_init(&re->ent, memhash(&re->cookie, sizeof(uint32_t)));
+ hashmap_add(&data->renames, &re->ent);
+}
+
+/*
+ * Handle directory renames
+ *
+ * Once a IN_MOVED_TO event is received, lookup the rename tracking information
+ * via the event cookie and use this information to update the watch.
+ */
+static void rename_dir(uint32_t cookie, const char *path,
+ struct fsm_listen_data *data)
+{
+ struct rename_entry rek, *re;
+ struct watch_entry k, *w;
+
+ /* lookup a pending rename to match */
+ rek.cookie = cookie;
+ hashmap_entry_init(&rek.ent, memhash(&rek.cookie, sizeof(uint32_t)));
+ re = hashmap_get_entry(&data->renames, &rek, ent, NULL);
+ if (re) {
+ k.dir = re->dir;
+ hashmap_entry_init(&k.ent, memhash(k.dir, strlen(k.dir)));
+ w = hashmap_get_entry(&data->revwatches, &k, ent, NULL);
+ if (w) {
+ w->cookie = 0; /* rename handled */
+ remove_watch(w, data);
+ add_watch(path, data);
+ } else {
+ BUG("No matching watch");
+ }
+ } else {
+ BUG("No matching cookie");
+ }
+}
+
+/*
+ * Recursively add watches to every directory under path
+ */
+static int register_inotify(const char *path, struct fsm_listen_data *data)
+{
+ DIR *dir;
+ struct strbuf current = STRBUF_INIT;
+ struct dirent *de;
+ struct stat fs;
+ int ret = -1;
+
+ dir = opendir(path);
+ if (!dir)
+ return error_errno("opendir('%s') failed", path);
+
+ while ((de = readdir_skip_dot_and_dotdot(dir)) != NULL) {
+ strbuf_reset(¤t);
+ strbuf_addf(¤t, "%s/%s", path, de->d_name);
+ if (lstat(current.buf, &fs)) {
+ error_errno("lstat('%s') failed", current.buf);
+ goto failed;
+ }
+
+ /* recurse into directory */
+ if (S_ISDIR(fs.st_mode)) {
+ if (add_watch(current.buf, data))
+ goto failed;
+ if (register_inotify(current.buf, data))
+ goto failed;
+ }
+ }
+ ret = 0;
+
+failed:
+ strbuf_release(¤t);
+ if (closedir(dir) < 0)
+ return error_errno("closedir('%s') failed", path);
+ return ret;
+}
+
+static int em_rename_dir_from(u_int32_t mask)
+{
+ return ((mask & IN_ISDIR) && (mask & IN_MOVED_FROM));
+}
+
+static int em_rename_dir_to(u_int32_t mask)
+{
+ return ((mask & IN_ISDIR) && (mask & IN_MOVED_TO));
+}
+
+static int em_remove_watch(u_int32_t mask)
+{
+ return (mask & IN_DELETE_SELF);
+}
+
+static int em_dir_renamed(u_int32_t mask)
+{
+ return ((mask & IN_ISDIR) && (mask & IN_MOVE));
+}
+
+static int em_dir_created(u_int32_t mask)
+{
+ return ((mask & IN_ISDIR) && (mask & IN_CREATE));
+}
+
+static int em_dir_deleted(uint32_t mask)
+{
+ return ((mask & IN_ISDIR) && (mask & IN_DELETE));
+}
+
+static int em_force_shutdown(u_int32_t mask)
+{
+ return (mask & IN_UNMOUNT) || (mask & IN_Q_OVERFLOW);
+}
+
+static int em_ignore(u_int32_t mask)
+{
+ return (mask & IN_IGNORED) || (mask & IN_MOVE_SELF);
+}
+
+static void log_mask_set(const char *path, u_int32_t mask)
+{
+ struct strbuf msg = STRBUF_INIT;
+
+ if (mask & IN_ACCESS)
+ strbuf_addstr(&msg, "IN_ACCESS|");
+ if (mask & IN_MODIFY)
+ strbuf_addstr(&msg, "IN_MODIFY|");
+ if (mask & IN_ATTRIB)
+ strbuf_addstr(&msg, "IN_ATTRIB|");
+ if (mask & IN_CLOSE_WRITE)
+ strbuf_addstr(&msg, "IN_CLOSE_WRITE|");
+ if (mask & IN_CLOSE_NOWRITE)
+ strbuf_addstr(&msg, "IN_CLOSE_NOWRITE|");
+ if (mask & IN_OPEN)
+ strbuf_addstr(&msg, "IN_OPEN|");
+ if (mask & IN_MOVED_FROM)
+ strbuf_addstr(&msg, "IN_MOVED_FROM|");
+ if (mask & IN_MOVED_TO)
+ strbuf_addstr(&msg, "IN_MOVED_TO|");
+ if (mask & IN_CREATE)
+ strbuf_addstr(&msg, "IN_CREATE|");
+ if (mask & IN_DELETE)
+ strbuf_addstr(&msg, "IN_DELETE|");
+ if (mask & IN_DELETE_SELF)
+ strbuf_addstr(&msg, "IN_DELETE_SELF|");
+ if (mask & IN_MOVE_SELF)
+ strbuf_addstr(&msg, "IN_MOVE_SELF|");
+ if (mask & IN_UNMOUNT)
+ strbuf_addstr(&msg, "IN_UNMOUNT|");
+ if (mask & IN_Q_OVERFLOW)
+ strbuf_addstr(&msg, "IN_Q_OVERFLOW|");
+ if (mask & IN_IGNORED)
+ strbuf_addstr(&msg, "IN_IGNORED|");
+ if (mask & IN_ISDIR)
+ strbuf_addstr(&msg, "IN_ISDIR|");
+
+ trace_printf_key(&trace_fsmonitor, "inotify_event: '%s', mask=%#8.8x %s",
+ path, mask, msg.buf);
+
+ strbuf_release(&msg);
+}
+
+int fsm_listen__ctor(struct fsmonitor_daemon_state *state)
+{
+ int fd;
+ int ret = 0;
+ struct fsm_listen_data *data;
+
+ CALLOC_ARRAY(data, 1);
+ state->listen_data = data;
+ state->listen_error_code = -1;
+ data->shutdown = SHUTDOWN_ERROR;
+
+ fd = inotify_init1(O_NONBLOCK);
+ if (fd < 0)
+ return error_errno("inotify_init1() failed");
+
+ data->fd_inotify = fd;
+
+ hashmap_init(&data->watches, watch_entry_cmp, NULL, 0);
+ hashmap_init(&data->renames, rename_entry_cmp, NULL, 0);
+ hashmap_init(&data->revwatches, revwatches_entry_cmp, NULL, 0);
+
+ if (add_watch(state->path_worktree_watch.buf, data))
+ ret = -1;
+ else if (register_inotify(state->path_worktree_watch.buf, data))
+ ret = -1;
+ else if (state->nr_paths_watching > 1) {
+ if (add_watch(state->path_gitdir_watch.buf, data))
+ ret = -1;
+ else if (register_inotify(state->path_gitdir_watch.buf, data))
+ ret = -1;
+ }
+
+ if (!ret) {
+ state->listen_error_code = 0;
+ data->shutdown = SHUTDOWN_CONTINUE;
+ }
+
+ return ret;
+}
+
+void fsm_listen__dtor(struct fsmonitor_daemon_state *state)
+{
+ struct fsm_listen_data *data;
+ struct hashmap_iter iter;
+ struct watch_entry *w;
+ int fd;
+
+ if (!state || !state->listen_data)
+ return;
+
+ data = state->listen_data;
+ fd = data->fd_inotify;
+
+ hashmap_for_each_entry(&data->watches, &iter, w, ent) {
+ w->cookie = 0; /* ignore any pending renames */
+ remove_watch(w, data);
+ }
+ hashmap_clear(&data->watches);
+
+ hashmap_clear(&data->revwatches); /* remove_watch freed the entries */
+
+ hashmap_clear_and_free(&data->renames, struct rename_entry, ent);
+
+ FREE_AND_NULL(state->listen_data);
+
+ if (fd && (close(fd) < 0))
+ error_errno(_("closing inotify file descriptor failed"));
+}
+
+void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)
+{
+ if (!state->listen_data->shutdown)
+ state->listen_data->shutdown = SHUTDOWN_STOP;
+}
+
+/*
+ * Process a single inotify event and queue for publication.
+ */
+static int process_event(const char *path,
+ const struct inotify_event *event,
+ struct fsmonitor_batch *batch,
+ struct string_list *cookie_list,
+ struct fsmonitor_daemon_state *state)
+{
+ const char *rel;
+ const char *last_sep;
+
+ switch (fsmonitor_classify_path_absolute(state, path)) {
+ case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:
+ case IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:
+ /* Use just the filename of the cookie file. */
+ last_sep = find_last_dir_sep(path);
+ string_list_append(cookie_list,
+ last_sep ? last_sep + 1 : path);
+ break;
+ case IS_INSIDE_DOT_GIT:
+ case IS_INSIDE_GITDIR:
+ break;
+ case IS_DOT_GIT:
+ case IS_GITDIR:
+ /*
+ * If .git directory is deleted or renamed away,
+ * we have to quit.
+ */
+ if (em_dir_deleted(event->mask)) {
+ trace_printf_key(&trace_fsmonitor,
+ "event: gitdir removed");
+ state->listen_data->shutdown = SHUTDOWN_FORCE;
+ goto done;
+ }
+
+ if (em_dir_renamed(event->mask)) {
+ trace_printf_key(&trace_fsmonitor,
+ "event: gitdir renamed");
+ state->listen_data->shutdown = SHUTDOWN_FORCE;
+ goto done;
+ }
+ break;
+ case IS_WORKDIR_PATH:
+ /* normal events in the working directory */
+ if (trace_pass_fl(&trace_fsmonitor))
+ log_mask_set(path, event->mask);
+
+ rel = path + state->path_worktree_watch.len + 1;
+ fsmonitor_batch__add_path(batch, rel);
+
+ if (em_dir_deleted(event->mask))
+ break;
+
+ /* received IN_MOVE_FROM, add tracking for expected IN_MOVE_TO */
+ if (em_rename_dir_from(event->mask))
+ add_dir_rename(event->cookie, path, state->listen_data);
+
+ /* received IN_MOVE_TO, update watch to reflect new path */
+ if (em_rename_dir_to(event->mask))
+ rename_dir(event->cookie, path, state->listen_data);
+
+ if (em_dir_created(event->mask)) {
+ if (add_watch(path, state->listen_data)) {
+ state->listen_data->shutdown = SHUTDOWN_ERROR;
+ goto done;
+ }
+ if (register_inotify(path, state->listen_data)) {
+ state->listen_data->shutdown = SHUTDOWN_ERROR;
+ goto done;
+ }
+ }
+ break;
+ case IS_OUTSIDE_CONE:
+ default:
+ trace_printf_key(&trace_fsmonitor,
+ "ignoring '%s'", path);
+ break;
+ }
+ return 0;
+done:
+ return -1;
+}
+
+/*
+ * Read the inotify event stream and pre-process events before further
+ * processing and eventual publishing.
+ */
+static void handle_events(struct fsmonitor_daemon_state *state)
+{
+ /* See https://man7.org/linux/man-pages/man7/inotify.7.html */
+ char buf[4096]
+ __attribute__ ((aligned(__alignof__(struct inotify_event))));
+
+ struct hashmap watches = state->listen_data->watches;
+ struct fsmonitor_batch *batch = NULL;
+ struct string_list cookie_list = STRING_LIST_INIT_DUP;
+ struct watch_entry k, *w;
+ struct strbuf path;
+ const struct inotify_event *event;
+ int fd = state->listen_data->fd_inotify;
+ ssize_t len;
+ char *ptr, *p;
+
+ strbuf_init(&path, PATH_MAX);
+
+ for(;;) {
+ len = read(fd, buf, sizeof(buf));
+ if (len == -1 && errno != EAGAIN) {
+ error_errno(_("reading inotify message stream failed"));
+ state->listen_data->shutdown = SHUTDOWN_ERROR;
+ goto done;
+ }
+
+ /* nothing to read */
+ if (len <= 0)
+ goto done;
+
+ /* Loop over all events in the buffer. */
+ for (ptr = buf; ptr < buf + len;
+ ptr += sizeof(struct inotify_event) + event->len) {
+
+ event = (const struct inotify_event *) ptr;
+
+ if (em_ignore(event->mask))
+ continue;
+
+ /* File system was unmounted or event queue overflowed */
+ if (em_force_shutdown(event->mask)) {
+ if (trace_pass_fl(&trace_fsmonitor))
+ log_mask_set("Forcing shutdown", event->mask);
+ state->listen_data->shutdown = SHUTDOWN_FORCE;
+ goto done;
+ }
+
+ hashmap_entry_init(&k.ent, memhash(&event->wd, sizeof(int)));
+ k.wd = event->wd;
+
+ w = hashmap_get_entry(&watches, &k, ent, NULL);
+ if (!w) /* should never happen */
+ BUG("No watch for '%s'", event->name);
+
+ /* directory watch was removed */
+ if (em_remove_watch(event->mask)) {
+ remove_watch(w, state->listen_data);
+ continue;
+ }
+
+ strbuf_reset(&path);
+ strbuf_add(&path, w->dir, strlen(w->dir));
+ strbuf_addch(&path, '/');
+ strbuf_add(&path, event->name, strlen(event->name));
+
+ p = fsmonitor__resolve_alias(path.buf, &state->alias);
+ if (!p)
+ p = strbuf_detach(&path, NULL);
+
+ if (!batch)
+ batch = fsmonitor_batch__new();
+
+ if (process_event(p, event, batch, &cookie_list, state)) {
+ free(p);
+ goto done;
+ }
+ free(p);
+ }
+ strbuf_reset(&path);
+ fsmonitor_publish(state, batch, &cookie_list);
+ string_list_clear(&cookie_list, 0);
+ batch = NULL;
+ }
+done:
+ strbuf_release(&path);
+ fsmonitor_batch__free_list(batch);
+ string_list_clear(&cookie_list, 0);
+}
+
+/*
+ * Non-blocking read of the inotify events stream. The inotify fd is polled
+ * frequently to help minimize the number of queue overflows.
+ */
+void fsm_listen__loop(struct fsmonitor_daemon_state *state)
+{
+ int poll_num;
+ const int interval = 1000;
+ time_t checked = time(NULL);
+ struct pollfd fds[1];
+ fds[0].fd = state->listen_data->fd_inotify;
+ fds[0].events = POLLIN;
+
+ for(;;) {
+ switch (state->listen_data->shutdown) {
+ case SHUTDOWN_CONTINUE:
+ poll_num = poll(fds, 1, 1);
+ if (poll_num == -1) {
+ if (errno == EINTR)
+ continue;
+ error_errno(_("polling inotify message stream failed"));
+ state->listen_data->shutdown = SHUTDOWN_ERROR;
+ continue;
+ }
+
+ if ((time(NULL) - checked) >= interval) {
+ checked = time(NULL);
+ if (check_stale_dir_renames(&state->listen_data->renames,
+ checked - interval)) {
+ trace_printf_key(&trace_fsmonitor,
+ "Missed IN_MOVED_TO events, forcing shutdown");
+ state->listen_data->shutdown = SHUTDOWN_FORCE;
+ continue;
+ }
+ }
+
+ if (poll_num > 0 && (fds[0].revents & POLLIN))
+ handle_events(state);
+
+ continue;
+ case SHUTDOWN_ERROR:
+ state->listen_error_code = -1;
+ ipc_server_stop_async(state->ipc_server_data);
+ break;
+ case SHUTDOWN_FORCE:
+ state->listen_error_code = 0;
+ ipc_server_stop_async(state->ipc_server_data);
+ break;
+ case SHUTDOWN_STOP:
+ default:
+ state->listen_error_code = 0;
+ break;
+ }
+ return;
+ }
+}
--
gitgitgadget
^ permalink raw reply related [flat|nested] 89+ messages in thread* Re: [PATCH v2 09/12] fsmonitor: implement filesystem change listener for Linux
2022-10-14 21:45 ` [PATCH v2 09/12] fsmonitor: implement filesystem change listener for Linux Eric DeCosta via GitGitGadget
@ 2022-10-18 11:59 ` Ævar Arnfjörð Bjarmason
0 siblings, 0 replies; 89+ messages in thread
From: Ævar Arnfjörð Bjarmason @ 2022-10-18 11:59 UTC (permalink / raw)
To: Eric DeCosta via GitGitGadget; +Cc: git, Eric Sunshine, Eric DeCosta
On Fri, Oct 14 2022, Eric DeCosta via GitGitGadget wrote:
> From: Eric DeCosta <edecosta@mathworks.com>
> [...]
> + strbuf_add(&path, w->dir, strlen(w->dir));
> + strbuf_addch(&path, '/');
> + strbuf_add(&path, event->name, strlen(event->name));
Don't do strbuf_add(&buf, x, strlen(x), just use strbuf_addstr(&buf, x)
instead.
The same goes for a couple of existing occurances that hit "master"
already in the just-merged fsmonitor topic, but in this case we can
change it in-flight still. Thanks!
^ permalink raw reply [flat|nested] 89+ messages in thread
* [PATCH v2 10/12] fsmonitor: enable fsmonitor for Linux
2022-10-14 21:45 ` [PATCH v2 " Eric DeCosta via GitGitGadget
` (8 preceding siblings ...)
2022-10-14 21:45 ` [PATCH v2 09/12] fsmonitor: implement filesystem change listener for Linux Eric DeCosta via GitGitGadget
@ 2022-10-14 21:45 ` Eric DeCosta via GitGitGadget
2022-10-14 21:45 ` [PATCH v2 11/12] fsmonitor: test updates Eric DeCosta via GitGitGadget
` (4 subsequent siblings)
14 siblings, 0 replies; 89+ messages in thread
From: Eric DeCosta via GitGitGadget @ 2022-10-14 21:45 UTC (permalink / raw)
To: git
Cc: Eric Sunshine, Ævar Arnfjörð Bjarmason,
Eric DeCosta, Eric DeCosta
From: Eric DeCosta <edecosta@mathworks.com>
Update build to enable fsmonitor for Linux.
Signed-off-by: Eric DeCosta <edecosta@mathworks.com>
---
config.mak.uname | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/config.mak.uname b/config.mak.uname
index 9f716cfba81..d8889f8570a 100644
--- a/config.mak.uname
+++ b/config.mak.uname
@@ -68,6 +68,15 @@ ifeq ($(uname_S),Linux)
ifneq ($(findstring .el7.,$(uname_R)),)
BASIC_CFLAGS += -std=c99
endif
+ # The builtin FSMonitor on Linux builds upon Simple-IPC. Both require
+ # Unix domain sockets and PThreads.
+ ifndef NO_PTHREADS
+ ifndef NO_UNIX_SOCKETS
+ FSMONITOR_DAEMON_BACKEND = linux
+ FSMONITOR_OS_SETTINGS = linux
+ FSMONITOR_DAEMON_COMMON = unix
+ endif
+ endif
endif
ifeq ($(uname_S),GNU/kFreeBSD)
HAVE_ALLOCA_H = YesPlease
--
gitgitgadget
^ permalink raw reply related [flat|nested] 89+ messages in thread* [PATCH v2 11/12] fsmonitor: test updates
2022-10-14 21:45 ` [PATCH v2 " Eric DeCosta via GitGitGadget
` (9 preceding siblings ...)
2022-10-14 21:45 ` [PATCH v2 10/12] fsmonitor: enable fsmonitor " Eric DeCosta via GitGitGadget
@ 2022-10-14 21:45 ` Eric DeCosta via GitGitGadget
2022-10-14 21:45 ` [PATCH v2 12/12] fsmonitor: update doc for Linux Eric DeCosta via GitGitGadget
` (3 subsequent siblings)
14 siblings, 0 replies; 89+ messages in thread
From: Eric DeCosta via GitGitGadget @ 2022-10-14 21:45 UTC (permalink / raw)
To: git
Cc: Eric Sunshine, Ævar Arnfjörð Bjarmason,
Eric DeCosta, Eric DeCosta
From: Eric DeCosta <edecosta@mathworks.com>
t7527-builtin-fsmonitor was leaking fsmonitor--daemon processes in some
cases.
Accomodate slight difference in the number of events generated on Linux.
On lower-powered systems, spin a little to give the daemon time
to respond to and log filesystem events.
Signed-off-by: Eric DeCosta <edecosta@mathworks.com>
---
t/t7527-builtin-fsmonitor.sh | 72 +++++++++++++++++++++++++++++-------
1 file changed, 58 insertions(+), 14 deletions(-)
diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh
index 1746d30cf6a..d27bd3662a7 100755
--- a/t/t7527-builtin-fsmonitor.sh
+++ b/t/t7527-builtin-fsmonitor.sh
@@ -13,7 +13,7 @@ fi
stop_daemon_delete_repo () {
r=$1 &&
test_might_fail git -C $r fsmonitor--daemon stop &&
- rm -rf $1
+ rm -rf $r
}
start_daemon () {
@@ -72,6 +72,32 @@ start_daemon () {
)
}
+IMPLICIT_TIMEOUT=5
+
+wait_for_update () {
+ func=$1 &&
+ file=$2 &&
+ sz=$(wc -c < "$file") &&
+ last=0 &&
+ $func &&
+ k=0 &&
+ while test "$k" -lt $IMPLICIT_TIMEOUT
+ do
+ nsz=$(wc -c < "$file")
+ if test "$nsz" -gt "$sz"
+ then
+ if test "$last" -eq "$nsz"
+ then
+ return 0
+ fi
+ last=$nsz
+ fi
+ sleep 1
+ k=$(( $k + 1 ))
+ done &&
+ return 0
+}
+
# Is a Trace2 data event present with the given catetory and key?
# We do not care what the value is.
#
@@ -137,7 +163,6 @@ test_expect_success 'implicit daemon start' '
# machines (where it might take a moment to wake and reschedule the
# daemon process) to avoid false alarms during test runs.)
#
-IMPLICIT_TIMEOUT=5
verify_implicit_shutdown () {
r=$1 &&
@@ -373,6 +398,10 @@ create_files () {
echo 3 >dir2/new
}
+rename_directory () {
+ mv dirtorename dirrenamed
+}
+
rename_files () {
mv rename renamed &&
mv dir1/rename dir1/renamed &&
@@ -427,10 +456,12 @@ test_expect_success 'edit some files' '
start_daemon --tf "$PWD/.git/trace" &&
- edit_files &&
+ wait_for_update edit_files "$PWD/.git/trace" &&
test-tool fsmonitor-client query --token 0 &&
+ test_might_fail git fsmonitor--daemon stop &&
+
grep "^event: dir1/modified$" .git/trace &&
grep "^event: dir2/modified$" .git/trace &&
grep "^event: modified$" .git/trace &&
@@ -442,10 +473,12 @@ test_expect_success 'create some files' '
start_daemon --tf "$PWD/.git/trace" &&
- create_files &&
+ wait_for_update create_files "$PWD/.git/trace" &&
test-tool fsmonitor-client query --token 0 &&
+ test_might_fail git fsmonitor--daemon stop &&
+
grep "^event: dir1/new$" .git/trace &&
grep "^event: dir2/new$" .git/trace &&
grep "^event: new$" .git/trace
@@ -456,10 +489,12 @@ test_expect_success 'delete some files' '
start_daemon --tf "$PWD/.git/trace" &&
- delete_files &&
+ wait_for_update delete_files "$PWD/.git/trace" &&
test-tool fsmonitor-client query --token 0 &&
+ test_might_fail git fsmonitor--daemon stop &&
+
grep "^event: dir1/delete$" .git/trace &&
grep "^event: dir2/delete$" .git/trace &&
grep "^event: delete$" .git/trace
@@ -470,10 +505,12 @@ test_expect_success 'rename some files' '
start_daemon --tf "$PWD/.git/trace" &&
- rename_files &&
+ wait_for_update rename_files "$PWD/.git/trace" &&
test-tool fsmonitor-client query --token 0 &&
+ test_might_fail git fsmonitor--daemon stop &&
+
grep "^event: dir1/rename$" .git/trace &&
grep "^event: dir2/rename$" .git/trace &&
grep "^event: rename$" .git/trace &&
@@ -487,10 +524,12 @@ test_expect_success 'rename directory' '
start_daemon --tf "$PWD/.git/trace" &&
- mv dirtorename dirrenamed &&
+ wait_for_update rename_directory "$PWD/.git/trace" &&
test-tool fsmonitor-client query --token 0 &&
+ test_might_fail git fsmonitor--daemon stop &&
+
grep "^event: dirtorename/*$" .git/trace &&
grep "^event: dirrenamed/*$" .git/trace
'
@@ -500,10 +539,12 @@ test_expect_success 'file changes to directory' '
start_daemon --tf "$PWD/.git/trace" &&
- file_to_directory &&
+ wait_for_update file_to_directory "$PWD/.git/trace" &&
test-tool fsmonitor-client query --token 0 &&
+ test_might_fail git fsmonitor--daemon stop &&
+
grep "^event: delete$" .git/trace &&
grep "^event: delete/new$" .git/trace
'
@@ -513,10 +554,12 @@ test_expect_success 'directory changes to a file' '
start_daemon --tf "$PWD/.git/trace" &&
- directory_to_file &&
+ wait_for_update directory_to_file "$PWD/.git/trace" &&
test-tool fsmonitor-client query --token 0 &&
+ test_might_fail git fsmonitor--daemon stop &&
+
grep "^event: dir1$" .git/trace
'
@@ -561,7 +604,7 @@ test_expect_success 'flush cached data' '
test-tool -C test_flush fsmonitor-client query --token "builtin:test_00000002:0" >actual_2 &&
nul_to_q <actual_2 >actual_q2 &&
- grep "^builtin:test_00000002:0Q$" actual_q2 &&
+ grep "^builtin:test_00000002:[0-1]Q$" actual_q2 &&
>test_flush/file_3 &&
@@ -732,7 +775,8 @@ u_values="$u1 $u2"
for u in $u_values
do
test_expect_success "unicode in repo root path: $u" '
- test_when_finished "stop_daemon_delete_repo $u" &&
+ test_when_finished \
+ "stop_daemon_delete_repo `echo "$u" | sed 's:x:\\\\\\\\\\\\\\x:g'`" &&
git init "$u" &&
echo 1 >"$u"/file1 &&
@@ -814,8 +858,7 @@ my_match_and_clean () {
}
test_expect_success 'submodule always visited' '
- test_when_finished "git -C super fsmonitor--daemon stop; \
- rm -rf super; \
+ test_when_finished "rm -rf super; \
rm -rf sub" &&
create_super super &&
@@ -883,7 +926,8 @@ have_t2_error_event () {
}
test_expect_success "stray submodule super-prefix warning" '
- test_when_finished "rm -rf super; \
+ test_when_finished "git -C super/dir_1/dir_2/sub fsmonitor--daemon stop; \
+ rm -rf super; \
rm -rf sub; \
rm super-sub.trace" &&
--
gitgitgadget
^ permalink raw reply related [flat|nested] 89+ messages in thread* [PATCH v2 12/12] fsmonitor: update doc for Linux
2022-10-14 21:45 ` [PATCH v2 " Eric DeCosta via GitGitGadget
` (10 preceding siblings ...)
2022-10-14 21:45 ` [PATCH v2 11/12] fsmonitor: test updates Eric DeCosta via GitGitGadget
@ 2022-10-14 21:45 ` Eric DeCosta via GitGitGadget
2022-10-14 23:32 ` [PATCH v2 00/12] fsmonitor: Implement fsmonitor " Junio C Hamano
` (2 subsequent siblings)
14 siblings, 0 replies; 89+ messages in thread
From: Eric DeCosta via GitGitGadget @ 2022-10-14 21:45 UTC (permalink / raw)
To: git
Cc: Eric Sunshine, Ævar Arnfjörð Bjarmason,
Eric DeCosta, Eric DeCosta
From: Eric DeCosta <edecosta@mathworks.com>
Update the documentation for Linux.
Signed-off-by: Eric DeCosta <edecosta@mathworks.com>
---
Documentation/config/fsmonitor--daemon.txt | 4 ++--
Documentation/git-fsmonitor--daemon.txt | 24 ++++++++++++++--------
2 files changed, 18 insertions(+), 10 deletions(-)
diff --git a/Documentation/config/fsmonitor--daemon.txt b/Documentation/config/fsmonitor--daemon.txt
index c225c6c9e74..2cafb040d96 100644
--- a/Documentation/config/fsmonitor--daemon.txt
+++ b/Documentation/config/fsmonitor--daemon.txt
@@ -4,8 +4,8 @@ fsmonitor.allowRemote::
behavior. Only respected when `core.fsmonitor` is set to `true`.
fsmonitor.socketDir::
- This Mac OS-specific option, if set, specifies the directory in
+ Mac OS and Linux-specific option. If set, specifies the directory in
which to create the Unix domain socket used for communication
between the fsmonitor daemon and various Git commands. The directory must
- reside on a native Mac OS filesystem. Only respected when `core.fsmonitor`
+ reside on a native filesystem. Only respected when `core.fsmonitor`
is set to `true`.
diff --git a/Documentation/git-fsmonitor--daemon.txt b/Documentation/git-fsmonitor--daemon.txt
index 8238eadb0e1..c2b08229c74 100644
--- a/Documentation/git-fsmonitor--daemon.txt
+++ b/Documentation/git-fsmonitor--daemon.txt
@@ -76,23 +76,31 @@ repositories; this may be overridden by setting `fsmonitor.allowRemote` to
correctly with all network-mounted repositories and such use is considered
experimental.
-On Mac OS, the inter-process communication (IPC) between various Git
+On Linux and Mac OS, the inter-process communication (IPC) between various Git
commands and the fsmonitor daemon is done via a Unix domain socket (UDS) -- a
-special type of file -- which is supported by native Mac OS filesystems,
-but not on network-mounted filesystems, NTFS, or FAT32. Other filesystems
-may or may not have the needed support; the fsmonitor daemon is not guaranteed
-to work with these filesystems and such use is considered experimental.
+special type of file -- which is supported by many native Linux and Mac OS
+filesystems, but not on network-mounted filesystems, NTFS, or FAT32. Other
+filesystems may or may not have the needed support; the fsmonitor daemon is not
+guaranteed to work with these filesystems and such use is considered
+experimental.
By default, the socket is created in the `.git` directory, however, if the
`.git` directory is on a network-mounted filesystem, it will be instead be
created at `$HOME/.git-fsmonitor-*` unless `$HOME` itself is on a
network-mounted filesystem in which case you must set the configuration
-variable `fsmonitor.socketDir` to the path of a directory on a Mac OS native
+variable `fsmonitor.socketDir` to the path of a directory on a native
filesystem in which to create the socket file.
If none of the above directories (`.git`, `$HOME`, or `fsmonitor.socketDir`)
-is on a native Mac OS file filesystem the fsmonitor daemon will report an
-error that will cause the daemon and the currently running command to exit.
+is on a native Linux or Mac OS filesystem the fsmonitor daemon will report
+an error that will cause the daemon to exit and the currently running command
+to issue a warning.
+
+On Linux, the fsmonitor daemon registers a watch for each directory in the
+repository. The default per-user limit for the number of watches on most Linux
+systems is 8192. This may not be sufficient for large repositories or if
+multiple instances of the fsmonitor daemon are running.
+See https://watchexec.github.io/docs/inotify-limits.html[Linux inotify limits] for more information.
CONFIGURATION
-------------
--
gitgitgadget
^ permalink raw reply related [flat|nested] 89+ messages in thread* Re: [PATCH v2 00/12] fsmonitor: Implement fsmonitor for Linux
2022-10-14 21:45 ` [PATCH v2 " Eric DeCosta via GitGitGadget
` (11 preceding siblings ...)
2022-10-14 21:45 ` [PATCH v2 12/12] fsmonitor: update doc for Linux Eric DeCosta via GitGitGadget
@ 2022-10-14 23:32 ` Junio C Hamano
2022-10-17 21:32 ` Eric DeCosta
2022-10-17 22:14 ` Glen Choo
2022-11-16 23:23 ` [PATCH v3 0/6] " Eric DeCosta via GitGitGadget
14 siblings, 1 reply; 89+ messages in thread
From: Junio C Hamano @ 2022-10-14 23:32 UTC (permalink / raw)
To: Eric DeCosta via GitGitGadget
Cc: git, Eric Sunshine, Ævar Arnfjörð Bjarmason,
Eric DeCosta
"Eric DeCosta via GitGitGadget" <gitgitgadget@gmail.com> writes:
> Goal is to deliver fsmonitor for Linux that is on par with fsmonitor for
> Windows and Mac OS.
>
> This patch set builds upon previous work for done for Windows and Mac OS
> (first 6 patches) to implement a fsmonitor back-end for Linux based on the
> Linux inotify API.
Again, the first six patches are a part of what is queued as
ed/fsmonitor-on-networked-macos that is now in 'next' but lacks a
fix-up commit from Jeff King.
I understand that it might not be easy/possible (e.g. perhaps it is
a limitation of GGG?), but I really prefer not to see them re-posted
as part of this series, as I have to apply them and make sure there
are no changes from the last one before discarding them.
Anyway, thanks for an update. Will requeue.
^ permalink raw reply [flat|nested] 89+ messages in thread* RE: [PATCH v2 00/12] fsmonitor: Implement fsmonitor for Linux
2022-10-14 23:32 ` [PATCH v2 00/12] fsmonitor: Implement fsmonitor " Junio C Hamano
@ 2022-10-17 21:32 ` Eric DeCosta
2022-10-17 22:22 ` Junio C Hamano
2022-10-18 14:02 ` Johannes Schindelin
0 siblings, 2 replies; 89+ messages in thread
From: Eric DeCosta @ 2022-10-17 21:32 UTC (permalink / raw)
To: Junio C Hamano, Eric DeCosta via GitGitGadget
Cc: git@vger.kernel.org, Eric Sunshine,
Ævar Arnfjörð Bjarmason
> -----Original Message-----
> From: Junio C Hamano <gitster@pobox.com>
> Sent: Friday, October 14, 2022 7:33 PM
> To: Eric DeCosta via GitGitGadget <gitgitgadget@gmail.com>
> Cc: git@vger.kernel.org; Eric Sunshine <sunshine@sunshineco.com>; Ævar
> Arnfjörð Bjarmason <avarab@gmail.com>; Eric DeCosta
> <edecosta@mathworks.com>
> Subject: Re: [PATCH v2 00/12] fsmonitor: Implement fsmonitor for Linux
>
> "Eric DeCosta via GitGitGadget" <gitgitgadget@gmail.com> writes:
>
> > Goal is to deliver fsmonitor for Linux that is on par with fsmonitor
> > for Windows and Mac OS.
> >
> > This patch set builds upon previous work for done for Windows and Mac
> > OS (first 6 patches) to implement a fsmonitor back-end for Linux based
> > on the Linux inotify API.
>
> Again, the first six patches are a part of what is queued as ed/fsmonitor-on-
> networked-macos that is now in 'next' but lacks a fix-up commit from Jeff
> King.
>
> I understand that it might not be easy/possible (e.g. perhaps it is a limitation
> of GGG?), but I really prefer not to see them re-posted as part of this series,
> as I have to apply them and make sure there are no changes from the last
> one before discarding them.
>
> Anyway, thanks for an update. Will requeue.
I looks like I can use GGG with the next branch, but I will have to open a new PR (and close the existing one).
-Eric
^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [PATCH v2 00/12] fsmonitor: Implement fsmonitor for Linux
2022-10-17 21:32 ` Eric DeCosta
@ 2022-10-17 22:22 ` Junio C Hamano
2022-10-18 14:02 ` Johannes Schindelin
1 sibling, 0 replies; 89+ messages in thread
From: Junio C Hamano @ 2022-10-17 22:22 UTC (permalink / raw)
To: Eric DeCosta
Cc: Eric DeCosta via GitGitGadget, git@vger.kernel.org, Eric Sunshine,
Ævar Arnfjörð Bjarmason
Eric DeCosta <edecosta@mathworks.com> writes:
>> I understand that it might not be easy/possible (e.g. perhaps it is a limitation
>> of GGG?), but I really prefer not to see them re-posted as part of this series,
>> as I have to apply them and make sure there are no changes from the last
>> one before discarding them.
>>
>> Anyway, thanks for an update. Will requeue.
>
> I looks like I can use GGG with the next branch, but I will have
> to open a new PR (and close the existing one).
Please don't.
The macOS topic (thanks for working on it, by the way) is about to
graduate to 'master', and the situation would probably "improve"
anyway when it happens, I hope.
Thanks.
^ permalink raw reply [flat|nested] 89+ messages in thread
* RE: [PATCH v2 00/12] fsmonitor: Implement fsmonitor for Linux
2022-10-17 21:32 ` Eric DeCosta
2022-10-17 22:22 ` Junio C Hamano
@ 2022-10-18 14:02 ` Johannes Schindelin
1 sibling, 0 replies; 89+ messages in thread
From: Johannes Schindelin @ 2022-10-18 14:02 UTC (permalink / raw)
To: Eric DeCosta
Cc: Junio C Hamano, Eric DeCosta via GitGitGadget,
git@vger.kernel.org, Eric Sunshine,
Ævar Arnfjörð Bjarmason
[-- Attachment #1: Type: text/plain, Size: 1958 bytes --]
Hi Eric,
On Mon, 17 Oct 2022, Eric DeCosta wrote:
> > -----Original Message-----
> > From: Junio C Hamano <gitster@pobox.com>
> > Sent: Friday, October 14, 2022 7:33 PM
> > To: Eric DeCosta via GitGitGadget <gitgitgadget@gmail.com>
> > Cc: git@vger.kernel.org; Eric Sunshine <sunshine@sunshineco.com>; Ævar
> > Arnfjörð Bjarmason <avarab@gmail.com>; Eric DeCosta
> > <edecosta@mathworks.com>
> > Subject: Re: [PATCH v2 00/12] fsmonitor: Implement fsmonitor for Linux
> >
> > "Eric DeCosta via GitGitGadget" <gitgitgadget@gmail.com> writes:
> >
> > > Goal is to deliver fsmonitor for Linux that is on par with fsmonitor
> > > for Windows and Mac OS.
> > >
> > > This patch set builds upon previous work for done for Windows and Mac
> > > OS (first 6 patches) to implement a fsmonitor back-end for Linux based
> > > on the Linux inotify API.
> >
> > Again, the first six patches are a part of what is queued as ed/fsmonitor-on-
> > networked-macos that is now in 'next' but lacks a fix-up commit from Jeff
> > King.
> >
> > I understand that it might not be easy/possible (e.g. perhaps it is a limitation
> > of GGG?), but I really prefer not to see them re-posted as part of this series,
> > as I have to apply them and make sure there are no changes from the last
> > one before discarding them.
GitGitGadget mirrors all branches from gitster/git to gitgitgadget/git, so
if you open a PR in the latter repository, you can use all of those
branches as targets.
But this PR is in git/git, which does not offer that.
> > Anyway, thanks for an update. Will requeue.
>
> I looks like I can use GGG with the next branch, but I will have to open a new PR (and close the existing one).
No, you do not have to open a new PR for that.
You just hit the Edit button on the top (as if you wanted to change the
title) and then select the target ("base") branch in the drop-down box
under the title.
Ciao,
Dscho
^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [PATCH v2 00/12] fsmonitor: Implement fsmonitor for Linux
2022-10-14 21:45 ` [PATCH v2 " Eric DeCosta via GitGitGadget
` (12 preceding siblings ...)
2022-10-14 23:32 ` [PATCH v2 00/12] fsmonitor: Implement fsmonitor " Junio C Hamano
@ 2022-10-17 22:14 ` Glen Choo
2022-10-18 4:17 ` Junio C Hamano
2022-10-20 15:43 ` Junio C Hamano
2022-11-16 23:23 ` [PATCH v3 0/6] " Eric DeCosta via GitGitGadget
14 siblings, 2 replies; 89+ messages in thread
From: Glen Choo @ 2022-10-17 22:14 UTC (permalink / raw)
To: Eric DeCosta via GitGitGadget, git
Cc: Eric Sunshine, Ævar Arnfjörð Bjarmason,
Eric DeCosta
At $DAYJOB, we observed that this topic breaks MacOS builds with sha1dc:
$ make NO_APPLE_COMMON_CRYPTO=1 DC_SHA1=1 NO_OPENSSL=1 compat/fsmonitor/fsm-ipc-darwin.o
CC compat/fsmonitor/fsm-ipc-darwin.o
compat/fsmonitor/fsm-ipc-darwin.c:13:2: error: unknown type name 'SHA_CTX'; did you mean 'SHA1_CTX'?
SHA_CTX sha1ctx;
^~~~~~~
SHA1_CTX
./sha1dc/sha1.h:55:3: note: 'SHA1_CTX' declared here
} SHA1_CTX;
^
compat/fsmonitor/fsm-ipc-darwin.c:16:21: error: use of undeclared identifier 'SHA_DIGEST_LENGTH'
unsigned char hash[SHA_DIGEST_LENGTH];
^
compat/fsmonitor/fsm-ipc-darwin.c:31:2: error: implicit declaration of function 'SHA1_Init' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
SHA1_Init(&sha1ctx);
^
compat/fsmonitor/fsm-ipc-darwin.c:31:2: note: did you mean 'SHA1DCInit'?
./sha1dc/sha1.h:58:6: note: 'SHA1DCInit' declared here
void SHA1DCInit(SHA1_CTX*);
^
compat/fsmonitor/fsm-ipc-darwin.c:32:2: error: implicit declaration of function 'SHA1_Update' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
SHA1_Update(&sha1ctx, r->worktree, strlen(r->worktree));
^
compat/fsmonitor/fsm-ipc-darwin.c:32:2: note: did you mean 'SHA1DCUpdate'?
./sha1dc/sha1.h:96:6: note: 'SHA1DCUpdate' declared here
void SHA1DCUpdate(SHA1_CTX*, const char*, size_t);
^
compat/fsmonitor/fsm-ipc-darwin.c:33:2: error: implicit declaration of function 'SHA1_Final' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
SHA1_Final(hash, &sha1ctx);
^
compat/fsmonitor/fsm-ipc-darwin.c:33:2: note: did you mean 'SHA1DCFinal'?
./sha1dc/sha1.h:100:6: note: 'SHA1DCFinal' declared here
int SHA1DCFinal(unsigned char[20], SHA1_CTX*);
^
5 errors generated.
make: *** [compat/fsmonitor/fsm-ipc-darwin.o] Error 1
Without NO_OPENSSL, this still fails, but with slightly different error
messages.
$ make NO_APPLE_COMMON_CRYPTO=1 DC_SHA1=1 compat/fsmonitor/fsm-ipc-darwin.o
CC compat/fsmonitor/fsm-ipc-darwin.o
compat/fsmonitor/fsm-ipc-darwin.c:31:2: error: 'SHA1_Init' is deprecated [-Werror,-Wdeprecated-declarations]
SHA1_Init(&sha1ctx);
^
/opt/local/include/openssl/sha.h:49:1: note: 'SHA1_Init' has been explicitly marked deprecated here
OSSL_DEPRECATEDIN_3_0 int SHA1_Init(SHA_CTX *c);
^
/opt/local/include/openssl/macros.h:182:49: note: expanded from macro 'OSSL_DEPRECATEDIN_3_0'
# define OSSL_DEPRECATEDIN_3_0 OSSL_DEPRECATED(3.0)
^
/opt/local/include/openssl/macros.h:62:52: note: expanded from macro 'OSSL_DEPRECATED'
# define OSSL_DEPRECATED(since) __attribute__((deprecated))
^
compat/fsmonitor/fsm-ipc-darwin.c:32:2: error: 'SHA1_Update' is deprecated [-Werror,-Wdeprecated-declarations]
SHA1_Update(&sha1ctx, r->worktree, strlen(r->worktree));
^
/opt/local/include/openssl/sha.h:50:1: note: 'SHA1_Update' has been explicitly marked deprecated here
OSSL_DEPRECATEDIN_3_0 int SHA1_Update(SHA_CTX *c, const void *data, size_t len);
^
/opt/local/include/openssl/macros.h:182:49: note: expanded from macro 'OSSL_DEPRECATEDIN_3_0'
# define OSSL_DEPRECATEDIN_3_0 OSSL_DEPRECATED(3.0)
^
/opt/local/include/openssl/macros.h:62:52: note: expanded from macro 'OSSL_DEPRECATED'
# define OSSL_DEPRECATED(since) __attribute__((deprecated))
^
compat/fsmonitor/fsm-ipc-darwin.c:33:2: error: 'SHA1_Final' is deprecated [-Werror,-Wdeprecated-declarations]
SHA1_Final(hash, &sha1ctx);
^
/opt/local/include/openssl/sha.h:51:1: note: 'SHA1_Final' has been explicitly marked deprecated here
OSSL_DEPRECATEDIN_3_0 int SHA1_Final(unsigned char *md, SHA_CTX *c);
^
/opt/local/include/openssl/macros.h:182:49: note: expanded from macro 'OSSL_DEPRECATEDIN_3_0'
# define OSSL_DEPRECATEDIN_3_0 OSSL_DEPRECATED(3.0)
^
/opt/local/include/openssl/macros.h:62:52: note: expanded from macro 'OSSL_DEPRECATED'
# define OSSL_DEPRECATED(since) __attribute__((deprecated))
^
3 errors generated.
make: *** [compat/fsmonitor/fsm-ipc-darwin.o] Error 1
"Eric DeCosta via GitGitGadget" <gitgitgadget@gmail.com> writes:
> Goal is to deliver fsmonitor for Linux that is on par with fsmonitor for
> Windows and Mac OS.
>
> This patch set builds upon previous work for done for Windows and Mac OS
> (first 6 patches) to implement a fsmonitor back-end for Linux based on the
> Linux inotify API. inotify differs significantly from the equivalent Windows
> and Mac OS APIs in that a watch must be registered for every directory of
> interest (rather than a singular watch at the root of the directory tree)
> and special care must be taken to handle directory renames correctly.
>
> More information about inotify:
> https://man7.org/linux/man-pages/man7/inotify.7.html
>
> v1 differs from v0:
>
> * Code review feedback
> * Update how and which code can be shared between Mac OS and Linux
> * Increase polling frequency to every 1ms (matches Mac OS)
> * Updates to t7527 to improve test stability
>
^ permalink raw reply [flat|nested] 89+ messages in thread* Re: [PATCH v2 00/12] fsmonitor: Implement fsmonitor for Linux
2022-10-17 22:14 ` Glen Choo
@ 2022-10-18 4:17 ` Junio C Hamano
2022-10-18 17:03 ` Glen Choo
` (2 more replies)
2022-10-20 15:43 ` Junio C Hamano
1 sibling, 3 replies; 89+ messages in thread
From: Junio C Hamano @ 2022-10-18 4:17 UTC (permalink / raw)
To: Glen Choo
Cc: Eric DeCosta via GitGitGadget, git, Eric Sunshine,
Ævar Arnfjörð Bjarmason, Eric DeCosta
Glen Choo <chooglen@google.com> writes:
> At $DAYJOB, we observed that this topic breaks MacOS builds with sha1dc:
Thanks for a report.
How dissapointing. The thing is that the topic has been in 'next'
since the 11th (i.e. early last week), and I know that you guys rely
on the tip of 'next' in working order to cut your internal releases,
but we did not hear about this until now. What makes it taste even
worse is that nobody else caught this, even though we seem to have a
couple of macOS jobs at GitHub Actions CI, there we didn't see any
breakage related to this.
Possible action items:
* See what configurations our two macOS jobs are using. If neither
is using sha1dc, I would say that is criminal [*] and at least
one of them should be updated to do so right away.
* The "two macOS CI jobs sharing too many configuration knobs" may
not be limited to just SHA-1 implementation, but unlike Linux
builds and tests, we may have similar "monoculture" issue in our
macOS CI builds. Those users, who depend on macOS port being
healthy, should help identify unnecessary overlaps between the
two, and more importantly, make sure we have CI builds with
configuration similar to what they actually use.
* Adding a few build-only-without-tests CI jobs also might help.
* Those who depend on working macOS port, especially those with
corporate backing who choose to use configurations that are
different from what we have CI builds for, are requested to
arrange a more frequent build test to catch a problem like this
much earlier.
Anything else I forgot that we can do to improve the situation? I
personally do not use macOS, I am not interested in using one, but
I do value those who choose to use macOS have happy git working on
their platform, so the stakeholders need to chip in.
Thanks.
[Footnote]
* Until the world migrates over to SHA-256, the collision detecting
SHA-1 implementation is what we must use unless there is a strong
reason not to. If we are not testing something that ought to be
the default, we are not doing a very good job.
^ permalink raw reply [flat|nested] 89+ messages in thread* Re: [PATCH v2 00/12] fsmonitor: Implement fsmonitor for Linux
2022-10-18 4:17 ` Junio C Hamano
@ 2022-10-18 17:03 ` Glen Choo
2022-10-18 17:11 ` Junio C Hamano
2022-10-19 1:19 ` Ævar Arnfjörð Bjarmason
2022-10-19 1:04 ` Ævar Arnfjörð Bjarmason
2022-10-20 16:13 ` Junio C Hamano
2 siblings, 2 replies; 89+ messages in thread
From: Glen Choo @ 2022-10-18 17:03 UTC (permalink / raw)
To: Junio C Hamano
Cc: Eric DeCosta via GitGitGadget, git, Eric Sunshine,
Ævar Arnfjörð Bjarmason, Eric DeCosta,
Johannes Schindelin
Cc-ed Johannes, who would know a lot more about CI than I do.
Junio C Hamano <gitster@pobox.com> writes:
> Glen Choo <chooglen@google.com> writes:
>
>> At $DAYJOB, we observed that this topic breaks MacOS builds with sha1dc:
>
> Thanks for a report.
>
> How dissapointing. The thing is that the topic has been in 'next'
> since the 11th (i.e. early last week), and I know that you guys rely
> on the tip of 'next' in working order to cut your internal releases,
> but we did not hear about this until now.
Yes. Unfortunately, we (Google's Git team) release on a weekly cadence;
we merge on Fridays and build on Mondays (PST), which makes this timing
extremely unfortunate.
> Possible action items:
>
> * See what configurations our two macOS jobs are using. If neither
> is using sha1dc, I would say that is criminal [*] and at least
> one of them should be updated to do so right away.
I'm not too familiar with the CI, but I took a quick peek at ci/lib.sh
and noticed that none of the jobs build with sha1dc, not even the Linux
or Windows ones, so..
> * The "two macOS CI jobs sharing too many configuration knobs" may
> not be limited to just SHA-1 implementation, but unlike Linux
> builds and tests, we may have similar "monoculture" issue in our
> macOS CI builds. Those users, who depend on macOS port being
> healthy, should help identify unnecessary overlaps between the
> two, and more importantly, make sure we have CI builds with
> configuration similar to what they actually use.
I don't think this is a macOS-specific issue; our CI just doesn't do a
good job with testing various build configurations.
> * Adding a few build-only-without-tests CI jobs also might help.
This sounds like the way to go IMO. It might be too expensive to run the
full test suite on every build configuration, but build-without-test
might be ok.
> * Those who depend on working macOS port, especially those with
> corporate backing who choose to use configurations that are
> different from what we have CI builds for, are requested to
> arrange a more frequent build test to catch a problem like this
> much earlier.
I wished we had caught it sooner too. The folks here generally agree
that our weekly release cycle is not ideal for reasons such as this.
Hopefully this is good motivation to move that work forward, though I
can't promise anything right now.
> Anything else I forgot that we can do to improve the situation? I
> personally do not use macOS, I am not interested in using one, but
> I do value those who choose to use macOS have happy git working on
> their platform, so the stakeholders need to chip in.
There's nothing else I can think of at the moment. Thanks for your
patience and for moving the conversation along.
>
> Thanks.
>
>
> [Footnote]
>
> * Until the world migrates over to SHA-256, the collision detecting
> SHA-1 implementation is what we must use unless there is a strong
> reason not to. If we are not testing something that ought to be
> the default, we are not doing a very good job.
^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [PATCH v2 00/12] fsmonitor: Implement fsmonitor for Linux
2022-10-18 17:03 ` Glen Choo
@ 2022-10-18 17:11 ` Junio C Hamano
2022-10-19 1:19 ` Ævar Arnfjörð Bjarmason
1 sibling, 0 replies; 89+ messages in thread
From: Junio C Hamano @ 2022-10-18 17:11 UTC (permalink / raw)
To: Glen Choo
Cc: Eric DeCosta via GitGitGadget, git, Eric Sunshine,
Ævar Arnfjörð Bjarmason, Eric DeCosta,
Johannes Schindelin
Glen Choo <chooglen@google.com> writes:
> I wished we had caught it sooner too. The folks here generally agree
> that our weekly release cycle is not ideal for reasons such as this.
> Hopefully this is good motivation to move that work forward, though I
> can't promise anything right now.
It is perfectly OK to have an automated trial build job that runs
more frequently than your weekly release cycle, though. It should
usually yield only a single bit of usable information (e.g. "there
is no 'does not even build from the source' issue in upstream") that
may give you assurance (e.g. "if we maintain the course, the next
real build for release would hopefully go smoothly"), but when it
breaks, you have more time to react.
Thanks.
^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [PATCH v2 00/12] fsmonitor: Implement fsmonitor for Linux
2022-10-18 17:03 ` Glen Choo
2022-10-18 17:11 ` Junio C Hamano
@ 2022-10-19 1:19 ` Ævar Arnfjörð Bjarmason
2022-10-19 2:28 ` Eric Sunshine
1 sibling, 1 reply; 89+ messages in thread
From: Ævar Arnfjörð Bjarmason @ 2022-10-19 1:19 UTC (permalink / raw)
To: Glen Choo
Cc: Junio C Hamano, Eric DeCosta via GitGitGadget, git, Eric Sunshine,
Eric DeCosta, Johannes Schindelin
On Tue, Oct 18 2022, Glen Choo wrote:
> Cc-ed Johannes, who would know a lot more about CI than I do.
>
> Junio C Hamano <gitster@pobox.com> writes:
>
>> Glen Choo <chooglen@google.com> writes:
>>
>>> At $DAYJOB, we observed that this topic breaks MacOS builds with sha1dc:
>>
>> Thanks for a report.
>>
>> How dissapointing. The thing is that the topic has been in 'next'
>> since the 11th (i.e. early last week), and I know that you guys rely
>> on the tip of 'next' in working order to cut your internal releases,
>> but we did not hear about this until now.
>
> Yes. Unfortunately, we (Google's Git team) release on a weekly cadence;
> we merge on Fridays and build on Mondays (PST), which makes this timing
> extremely unfortunate.
>
>> Possible action items:
>>
>> * See what configurations our two macOS jobs are using. If neither
>> is using sha1dc, I would say that is criminal [*] and at least
>> one of them should be updated to do so right away.
>
> I'm not too familiar with the CI, but I took a quick peek at ci/lib.sh
> and noticed that none of the jobs build with sha1dc, not even the Linux
> or Windows ones, so..
All of our jobs except the OSX one build with SHA1_DC, because it's the
default.
Per my just-sent
https://lore.kernel.org/git/cover-v2-0.4-00000000000-20221019T010222Z-avarab@gmail.com/
the blind spot has been lack fo SHA1_DC on OSX, for others it's the
reverse, we don't test e.g. BLK_SHA1.
In practice we've been catching SHA-implementation specific code early
because the OSX implementation was different, but in this case it's
OSX-only code, so it only supported the Apple Common Crypto backend.
>> * The "two macOS CI jobs sharing too many configuration knobs" may
>> not be limited to just SHA-1 implementation, but unlike Linux
>> builds and tests, we may have similar "monoculture" issue in our
>> macOS CI builds. Those users, who depend on macOS port being
>> healthy, should help identify unnecessary overlaps between the
>> two, and more importantly, make sure we have CI builds with
>> configuration similar to what they actually use.
>
> I don't think this is a macOS-specific issue; our CI just doesn't do a
> good job with testing various build configurations.
Yes, definitely.
>> * Adding a few build-only-without-tests CI jobs also might help.
>
> This sounds like the way to go IMO. It might be too expensive to run the
> full test suite on every build configuration, but build-without-test
> might be ok.
Yes, there's not much point in running the full tests for some of these
variants.
A lot of it we can also get for free, e.g. we run some linux jobs with
clang, some with gcc etc., we could also run one with BLK_SHA1, one with
OPENSSL_SHA1 etc.
^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [PATCH v2 00/12] fsmonitor: Implement fsmonitor for Linux
2022-10-19 1:19 ` Ævar Arnfjörð Bjarmason
@ 2022-10-19 2:28 ` Eric Sunshine
2022-10-19 16:58 ` Junio C Hamano
2022-10-19 19:11 ` Ævar Arnfjörð Bjarmason
0 siblings, 2 replies; 89+ messages in thread
From: Eric Sunshine @ 2022-10-19 2:28 UTC (permalink / raw)
To: Ævar Arnfjörð Bjarmason
Cc: Glen Choo, Junio C Hamano, Eric DeCosta via GitGitGadget, git,
Eric DeCosta, Johannes Schindelin
On Tue, Oct 18, 2022 at 9:22 PM Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
> On Tue, Oct 18 2022, Glen Choo wrote:
> > I'm not too familiar with the CI, but I took a quick peek at ci/lib.sh
> > and noticed that none of the jobs build with sha1dc, not even the Linux
> > or Windows ones, so..
>
> All of our jobs except the OSX one build with SHA1_DC, because it's the
> default.
>
> Per my just-sent
> https://lore.kernel.org/git/cover-v2-0.4-00000000000-20221019T010222Z-avarab@gmail.com/
> the blind spot has been lack fo SHA1_DC on OSX, for others it's the
> reverse, we don't test e.g. BLK_SHA1.
>
> In practice we've been catching SHA-implementation specific code early
> because the OSX implementation was different, but in this case it's
> OSX-only code, so it only supported the Apple Common Crypto backend.
I don't know how germane it is to the current thread, but previous
discussions[1,2,3,4] favored dropping use of Apple's Common Crypto
altogether since it doesn't seem to buy us much (or anything) and is
incomplete; it doesn't support all of the OpenSSL API Git uses.
[1]: https://lore.kernel.org/git/CAPig+cTfMx_kwUAxBRHp6kNSOtXsdsv=odUQSRYVpV21DnRuvA@mail.gmail.com/
[2]: https://lore.kernel.org/git/CAMYxyaVQyVRQb-b0nVv412tMZ3rEnOfUPRakg2dEREg5_Ba5Ag@mail.gmail.com/T/
[3]: https://lore.kernel.org/git/20160102234923.GA14424@gmail.com/
[4]: https://lore.kernel.org/git/CAPig+cQ5kKAt2_RQnqT7Rn=uGmHV9VvxpQ+UgDPOj=D=pq6arg@mail.gmail.com/
^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [PATCH v2 00/12] fsmonitor: Implement fsmonitor for Linux
2022-10-19 2:28 ` Eric Sunshine
@ 2022-10-19 16:58 ` Junio C Hamano
2022-10-19 19:11 ` Ævar Arnfjörð Bjarmason
1 sibling, 0 replies; 89+ messages in thread
From: Junio C Hamano @ 2022-10-19 16:58 UTC (permalink / raw)
To: Eric Sunshine
Cc: Ævar Arnfjörð Bjarmason, Glen Choo,
Eric DeCosta via GitGitGadget, git, Eric DeCosta,
Johannes Schindelin
Eric Sunshine <sunshine@sunshineco.com> writes:
> I don't know how germane it is to the current thread, but previous
> discussions[1,2,3,4] favored dropping use of Apple's Common Crypto
> altogether since it doesn't seem to buy us much (or anything) and is
> incomplete; it doesn't support all of the OpenSSL API Git uses.
Yeah, that matches my recollection. Unless the situation has much
changed, dropping it may not be a bad thing to do.
But
* Fixing the fsmonitor code so that it can also be used with things
other than Common Crypto is the most urgent. The topic gave us a
grave regression (those who used to successfully build Git can no
longer build with their favourite configuration).
* Updating the build procedure so that sha1dc is used by default
everywhere is a good idea, but that is less urgent and should be
done separately, preferrably long after the dust settles from the
above.
* Removing Common Crypto support may not be a bad idea, but that is
even less urgent, unless the support burden is slowing us down or
forcing us to settle on common set of features that is too
limiting.
Thanks.
^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [PATCH v2 00/12] fsmonitor: Implement fsmonitor for Linux
2022-10-19 2:28 ` Eric Sunshine
2022-10-19 16:58 ` Junio C Hamano
@ 2022-10-19 19:11 ` Ævar Arnfjörð Bjarmason
2022-10-19 20:14 ` Eric Sunshine
1 sibling, 1 reply; 89+ messages in thread
From: Ævar Arnfjörð Bjarmason @ 2022-10-19 19:11 UTC (permalink / raw)
To: Eric Sunshine
Cc: Glen Choo, Junio C Hamano, Eric DeCosta via GitGitGadget, git,
Eric DeCosta, Johannes Schindelin
On Tue, Oct 18 2022, Eric Sunshine wrote:
> On Tue, Oct 18, 2022 at 9:22 PM Ævar Arnfjörð Bjarmason
> <avarab@gmail.com> wrote:
>> On Tue, Oct 18 2022, Glen Choo wrote:
>> > I'm not too familiar with the CI, but I took a quick peek at ci/lib.sh
>> > and noticed that none of the jobs build with sha1dc, not even the Linux
>> > or Windows ones, so..
>>
>> All of our jobs except the OSX one build with SHA1_DC, because it's the
>> default.
>>
>> Per my just-sent
>> https://lore.kernel.org/git/cover-v2-0.4-00000000000-20221019T010222Z-avarab@gmail.com/
>> the blind spot has been lack fo SHA1_DC on OSX, for others it's the
>> reverse, we don't test e.g. BLK_SHA1.
>>
>> In practice we've been catching SHA-implementation specific code early
>> because the OSX implementation was different, but in this case it's
>> OSX-only code, so it only supported the Apple Common Crypto backend.
>
> I don't know how germane it is to the current thread, but previous
> discussions[1,2,3,4] favored dropping use of Apple's Common Crypto
> altogether since it doesn't seem to buy us much (or anything) and is
> incomplete; it doesn't support all of the OpenSSL API Git uses.
>
> [1]: https://lore.kernel.org/git/CAPig+cTfMx_kwUAxBRHp6kNSOtXsdsv=odUQSRYVpV21DnRuvA@mail.gmail.com/
> [2]: https://lore.kernel.org/git/CAMYxyaVQyVRQb-b0nVv412tMZ3rEnOfUPRakg2dEREg5_Ba5Ag@mail.gmail.com/T/
> [3]: https://lore.kernel.org/git/20160102234923.GA14424@gmail.com/
> [4]: https://lore.kernel.org/git/CAPig+cQ5kKAt2_RQnqT7Rn=uGmHV9VvxpQ+UgDPOj=D=pq6arg@mail.gmail.com/
Yeah, maybe. I'd be 100% OK with that happening, but I don't use OSX
outside fo CI really, so I wanted to avoid scope creep to "non-SHA stuff
we use OpenSSL/AppleCommonCrypto/whatever" for, in the cases where we
can/do use OpenSSL/AppleCommonCrypto/whatever for SHA also.
I.e. that's basically a question about what https://, git-imap-send
etc. should be using if not vanilla OpenSSL & the like.
But if you/someone can come up with a patch that confidently asserts
that it's useless & drops it I'd be happy to either integrate it &
submit it as part of this series if it makes things simpler, or
alternatively re-roll on top of it.
I'm just not confident in making that case myself, since I hardly use
the platform, am not too familiar with the various concerns (aside from
skimming the links above), and don't really have interest in pursuing
that.
OTOH if you do want to make the case for dropping Apple Common Crypto
that would also presumably be much easier after this series, once we're
past justifying the hurdle of "wait, we're switching the thing we use
for SHA-1 by default, which we build practically everything in git on?"
:)
1. https://lore.kernel.org/git/CAPig+cQ5kKAt2_RQnqT7Rn=uGmHV9VvxpQ+UgDPOj=D=pq6arg@mail.gmail.com/
^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [PATCH v2 00/12] fsmonitor: Implement fsmonitor for Linux
2022-10-19 19:11 ` Ævar Arnfjörð Bjarmason
@ 2022-10-19 20:14 ` Eric Sunshine
0 siblings, 0 replies; 89+ messages in thread
From: Eric Sunshine @ 2022-10-19 20:14 UTC (permalink / raw)
To: Ævar Arnfjörð Bjarmason
Cc: Glen Choo, Junio C Hamano, Eric DeCosta via GitGitGadget, git,
Eric DeCosta, Johannes Schindelin
On Wed, Oct 19, 2022 at 3:16 PM Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
> On Tue, Oct 18 2022, Eric Sunshine wrote:
> > On Tue, Oct 18, 2022 at 9:22 PM Ævar Arnfjörð Bjarmason
> > <avarab@gmail.com> wrote:
> >> In practice we've been catching SHA-implementation specific code early
> >> because the OSX implementation was different, but in this case it's
> >> OSX-only code, so it only supported the Apple Common Crypto backend.
> >
> > I don't know how germane it is to the current thread, but previous
> > discussions[1,2,3,4] favored dropping use of Apple's Common Crypto
> > altogether since it doesn't seem to buy us much (or anything) and is
> > incomplete; it doesn't support all of the OpenSSL API Git uses.
>
> Yeah, maybe. I'd be 100% OK with that happening, but I don't use OSX
> outside fo CI really, so I wanted to avoid scope creep to "non-SHA stuff
> we use OpenSSL/AppleCommonCrypto/whatever" for, in the cases where we
> can/do use OpenSSL/AppleCommonCrypto/whatever for SHA also.
Indeed, I was not suggesting that retirement of AppleCommonCrypto be
tackled by you or by the series you posted to fix the build failure;
doing so is well outside the scope of the immediate problem. I brought
up those old discussions only as reference and as a pointer that
whatever fix is eventually adopted for the immediate build problem
need not necessarily bend over backward for AppleCommonCrypto support
if a long-term goal is to drop AppleCommonCrypto (i.e. do as little as
necessary to keep AppleCommonCrypto in a working state, but don't go
overboard trying to give it first-class support since it will never be
a complete replacement for OpenSSL).
> But if you/someone can come up with a patch that confidently asserts
> that it's useless & drops it I'd be happy to either integrate it &
> submit it as part of this series if it makes things simpler, or
> alternatively re-roll on top of it.
>
> I'm just not confident in making that case myself, since I hardly use
> the platform, am not too familiar with the various concerns (aside from
> skimming the links above), and don't really have interest in pursuing
> that.
I was not suggesting holding up your series or integrating retirement
of AppleCommonCrypto with it; they are separate concerns.
> OTOH if you do want to make the case for dropping Apple Common Crypto
> that would also presumably be much easier after this series, once we're
> past justifying the hurdle of "wait, we're switching the thing we use
> for SHA-1 by default, which we build practically everything in git on?"
My "don't know how germane it is to the current thread" observation
was in response to the `fsmonitor` thread, before I ever saw the
series you posted for fixing the build failure, so it wasn't given as
any sort of feedback on your series, and wasn't asking you to
incorporate retirement of AppleCommonCrypto into your series. That
said, it does appear that dropping AppleCommonCrypto should become
easier after your changes land if someone opts to tackle the task.
^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [PATCH v2 00/12] fsmonitor: Implement fsmonitor for Linux
2022-10-18 4:17 ` Junio C Hamano
2022-10-18 17:03 ` Glen Choo
@ 2022-10-19 1:04 ` Ævar Arnfjörð Bjarmason
2022-10-19 16:33 ` Junio C Hamano
2022-10-20 16:13 ` Junio C Hamano
2 siblings, 1 reply; 89+ messages in thread
From: Ævar Arnfjörð Bjarmason @ 2022-10-19 1:04 UTC (permalink / raw)
To: Junio C Hamano
Cc: Glen Choo, Eric DeCosta via GitGitGadget, git, Eric Sunshine,
Eric DeCosta
On Mon, Oct 17 2022, Junio C Hamano wrote:
> Glen Choo <chooglen@google.com> writes:
>
>> At $DAYJOB, we observed that this topic breaks MacOS builds with sha1dc:
>
> Thanks for a report.
>
> How dissapointing. The thing is that the topic has been in 'next'
> since the 11th (i.e. early last week), and I know that you guys rely
> on the tip of 'next' in working order to cut your internal releases,
> but we did not hear about this until now. What makes it taste even
> worse is that nobody else caught this, even though we seem to have a
> couple of macOS jobs at GitHub Actions CI, there we didn't see any
> breakage related to this.
FWIW I see you caught it on the 9th in
https://lore.kernel.org/git/xmqqh70c62w0.fsf@gitster.g/, but then the
base topic was merged down on the 17th.
> Possible action items:
>
> * See what configurations our two macOS jobs are using. If neither
> is using sha1dc, I would say that is criminal [*] and at least
> one of them should be updated to do so right away.
I submitted a v2 of my series to finally make OSX use SHA1DC by default:
https://lore.kernel.org/git/cover-v2-0.4-00000000000-20221019T010222Z-avarab@gmail.com/
As part of that we'll CI that & the current "apple common crypto"
implementations, currently we just do the latter, which is why we didn't
catch this.
We could follow-up and CI the OpenSSL one too, but that was a larger
change, so I punted on it.
In any case,
https://lore.kernel.org/git/patch-v2-1.4-392fabdb456-20221019T010222Z-avarab@gmail.com/
in that series un-breaks master, maybe you're interested in peeling it
off & fast-tracking it? I didn't submit it separately because I don't
know how much of a rush we're in there (given that CI isn't broken).
^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [PATCH v2 00/12] fsmonitor: Implement fsmonitor for Linux
2022-10-19 1:04 ` Ævar Arnfjörð Bjarmason
@ 2022-10-19 16:33 ` Junio C Hamano
0 siblings, 0 replies; 89+ messages in thread
From: Junio C Hamano @ 2022-10-19 16:33 UTC (permalink / raw)
To: Ævar Arnfjörð Bjarmason
Cc: Glen Choo, Eric DeCosta via GitGitGadget, git, Eric Sunshine,
Eric DeCosta
Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:
> On Mon, Oct 17 2022, Junio C Hamano wrote:
>
>> Glen Choo <chooglen@google.com> writes:
>>
>>> At $DAYJOB, we observed that this topic breaks MacOS builds with sha1dc:
>>
>> Thanks for a report.
>>
>> How dissapointing. The thing is that the topic has been in 'next'
>> since the 11th (i.e. early last week), and I know that you guys rely
>> on the tip of 'next' in working order to cut your internal releases,
>> but we did not hear about this until now. What makes it taste even
>> worse is that nobody else caught this, even though we seem to have a
>> couple of macOS jobs at GitHub Actions CI, there we didn't see any
>> breakage related to this.
>
> FWIW I see you caught it on the 9th in
> https://lore.kernel.org/git/xmqqh70c62w0.fsf@gitster.g/, but then the
> base topic was merged down on the 17th.
Heh, I already forgot I made that comment myself ;-) Thanks for
reminding, and thanks for picking the single step to fast-track the
fix.
^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [PATCH v2 00/12] fsmonitor: Implement fsmonitor for Linux
2022-10-18 4:17 ` Junio C Hamano
2022-10-18 17:03 ` Glen Choo
2022-10-19 1:04 ` Ævar Arnfjörð Bjarmason
@ 2022-10-20 16:13 ` Junio C Hamano
2022-10-20 16:37 ` Eric Sunshine
2 siblings, 1 reply; 89+ messages in thread
From: Junio C Hamano @ 2022-10-20 16:13 UTC (permalink / raw)
To: Glen Choo
Cc: Eric DeCosta via GitGitGadget, git, Eric Sunshine,
Ævar Arnfjörð Bjarmason, Eric DeCosta
Junio C Hamano <gitster@pobox.com> writes:
> Possible action items:
>
> * See what configurations our two macOS jobs are using. If neither
> is using sha1dc, I would say that is criminal [*] and at least
> one of them should be updated to do so right away.
So here is my "panda-see-panda-do" attempt.
----- >8 --------- >8 --------- >8 --------- >8 --------- >8 -----
Subject: [PATCH] ci: use DC_SHA1=1 on osx-clang job for CI
All other jobs were using the default DC_SHA1 (which is a
recommended practice), but the default for macOS jobs being Apple's
common crypt, we didn't catch recent breakage soon enough.
We may want to give similar diversity for Linux jobs so that some of
them build with other implementations of SHA-1, but let's start
small and fill only the immediate need.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
ci/lib.sh | 2 ++
1 file changed, 2 insertions(+)
diff --git a/ci/lib.sh b/ci/lib.sh
index 1b0cc2b57d..5a115704cb 100755
--- a/ci/lib.sh
+++ b/ci/lib.sh
@@ -259,6 +259,8 @@ macos-latest)
MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=$(which python3)"
else
MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=$(which python2)"
+ MAKEFLAGS="$MAKEFLAGS NO_APPLE_COMMON_CRYPTO=NoThanks"
+ MAKEFLAGS="$MAKEFLAGS DC_SHA1=1 NO_OPENSSL=1"
fi
;;
esac
--
2.38.1-267-g82836053dd
^ permalink raw reply related [flat|nested] 89+ messages in thread
* Re: [PATCH v2 00/12] fsmonitor: Implement fsmonitor for Linux
2022-10-20 16:13 ` Junio C Hamano
@ 2022-10-20 16:37 ` Eric Sunshine
2022-10-20 17:01 ` Junio C Hamano
0 siblings, 1 reply; 89+ messages in thread
From: Eric Sunshine @ 2022-10-20 16:37 UTC (permalink / raw)
To: Junio C Hamano
Cc: Glen Choo, Eric DeCosta via GitGitGadget, git,
Ævar Arnfjörð Bjarmason, Eric DeCosta
On Thu, Oct 20, 2022 at 12:13 PM Junio C Hamano <gitster@pobox.com> wrote:
> So here is my "panda-see-panda-do" attempt.
>
> ----- >8 --------- >8 --------- >8 --------- >8 --------- >8 -----
> Subject: [PATCH] ci: use DC_SHA1=1 on osx-clang job for CI
>
> All other jobs were using the default DC_SHA1 (which is a
> recommended practice), but the default for macOS jobs being Apple's
> common crypt, we didn't catch recent breakage soon enough.
"recent breakage" is quite vague and probably won't help future
readers understand what this is actually fixing. Possible
improvements: (1) a prose description of the breakage; (2) the actual
compiler error message; (3) a pointer[1] to the email reporting the
problem. One or more of the above improvements to the commit message
would help future readers.
[1]: https://lore.kernel.org/git/kl6l7d0yyu6r.fsf@chooglen-macbookpro.roam.corp.google.com/
> We may want to give similar diversity for Linux jobs so that some of
> them build with other implementations of SHA-1, but let's start
> small and fill only the immediate need.
>
> Signed-off-by: Junio C Hamano <gitster@pobox.com>
> ---
> diff --git a/ci/lib.sh b/ci/lib.sh
> index 1b0cc2b57d..5a115704cb 100755
> --- a/ci/lib.sh
> +++ b/ci/lib.sh
> @@ -259,6 +259,8 @@ macos-latest)
> MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=$(which python3)"
> else
> MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=$(which python2)"
> + MAKEFLAGS="$MAKEFLAGS NO_APPLE_COMMON_CRYPTO=NoThanks"
> + MAKEFLAGS="$MAKEFLAGS DC_SHA1=1 NO_OPENSSL=1"
> fi
> ;;
> esac
^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [PATCH v2 00/12] fsmonitor: Implement fsmonitor for Linux
2022-10-20 16:37 ` Eric Sunshine
@ 2022-10-20 17:01 ` Junio C Hamano
2022-10-20 17:54 ` Eric Sunshine
0 siblings, 1 reply; 89+ messages in thread
From: Junio C Hamano @ 2022-10-20 17:01 UTC (permalink / raw)
To: Eric Sunshine
Cc: Glen Choo, Eric DeCosta via GitGitGadget, git,
Ævar Arnfjörð Bjarmason, Eric DeCosta
Eric Sunshine <sunshine@sunshineco.com> writes:
> On Thu, Oct 20, 2022 at 12:13 PM Junio C Hamano <gitster@pobox.com> wrote:
>> So here is my "panda-see-panda-do" attempt.
>>
>> ----- >8 --------- >8 --------- >8 --------- >8 --------- >8 -----
>> Subject: [PATCH] ci: use DC_SHA1=1 on osx-clang job for CI
>>
>> All other jobs were using the default DC_SHA1 (which is a
>> recommended practice), but the default for macOS jobs being Apple's
>> common crypt, we didn't catch recent breakage soon enough.
>
> "recent breakage" is quite vague and probably won't help future
> readers understand what this is actually fixing. Possible
> improvements: (1) a prose description of the breakage; (2) the actual
> compiler error message; (3) a pointer[1] to the email reporting the
> problem. One or more of the above improvements to the commit message
> would help future readers.
I do not think (2) or (3) would help all that much. A finger that
points at the exact commit that broke the build (with the condition
under which the build breaks) would probably be the most useful to
those who read "git log" output.
Thanks.
----- >8 --------- >8 --------- >8 --------- >8 --------- >8 -----
Subject: [PATCH] ci: use DC_SHA1=YesPlease on osx-clang job for CI
7b8cfe34 (Merge branch 'ed/fsmonitor-on-networked-macos',
2022-10-17) broke the build on macOS with sha1dc by bypassing our
hash abstraction (git_SHA_CTX etc.), but it wasn't caught before the
problematic topic was merged down to the 'master' branch. Nobody
was even compile testing with DC_SHA1 set, although it is the
recommended choice in these days for folks when they use SHA-1.
This was because the default for macOS uses Apple Common Crypto, and
both of the two CI jobs did not override the default. Tweak one of
them to use DC_SHA1 to improve the coverage.
We may want to give similar diversity for Linux jobs so that some of
them build with other implementations of SHA-1; they currently all
build and test with DC_SHA1 as that is the default on everywhere
other than macOS.
But let's start small to fill only the immediate need.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
ci/lib.sh | 2 ++
1 file changed, 2 insertions(+)
diff --git a/ci/lib.sh b/ci/lib.sh
index 1b0cc2b57d..51e47aa5d6 100755
--- a/ci/lib.sh
+++ b/ci/lib.sh
@@ -259,6 +259,8 @@ macos-latest)
MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=$(which python3)"
else
MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=$(which python2)"
+ MAKEFLAGS="$MAKEFLAGS NO_APPLE_COMMON_CRYPTO=NoThanks"
+ MAKEFLAGS="$MAKEFLAGS DC_SHA1=YesPlease NO_OPENSSL=NoThanks"
fi
;;
esac
--
2.38.1-271-ge78cd6c470
^ permalink raw reply related [flat|nested] 89+ messages in thread
* Re: [PATCH v2 00/12] fsmonitor: Implement fsmonitor for Linux
2022-10-20 17:01 ` Junio C Hamano
@ 2022-10-20 17:54 ` Eric Sunshine
0 siblings, 0 replies; 89+ messages in thread
From: Eric Sunshine @ 2022-10-20 17:54 UTC (permalink / raw)
To: Junio C Hamano
Cc: Glen Choo, Eric DeCosta via GitGitGadget, git,
Ævar Arnfjörð Bjarmason, Eric DeCosta
On Thu, Oct 20, 2022 at 1:01 PM Junio C Hamano <gitster@pobox.com> wrote:
> Eric Sunshine <sunshine@sunshineco.com> writes:
> > On Thu, Oct 20, 2022 at 12:13 PM Junio C Hamano <gitster@pobox.com> wrote:
> >> All other jobs were using the default DC_SHA1 (which is a
> >> recommended practice), but the default for macOS jobs being Apple's
> >> common crypt, we didn't catch recent breakage soon enough.
> >
> > "recent breakage" is quite vague and probably won't help future
> > readers understand what this is actually fixing. Possible
> > improvements: (1) a prose description of the breakage; (2) the actual
> > compiler error message; (3) a pointer[1] to the email reporting the
> > problem. One or more of the above improvements to the commit message
> > would help future readers.
>
> I do not think (2) or (3) would help all that much. A finger that
> points at the exact commit that broke the build (with the condition
> under which the build breaks) would probably be the most useful to
> those who read "git log" output.
>
> ----- >8 --------- >8 --------- >8 --------- >8 --------- >8 -----
> Subject: [PATCH] ci: use DC_SHA1=YesPlease on osx-clang job for CI
>
> 7b8cfe34 (Merge branch 'ed/fsmonitor-on-networked-macos',
> 2022-10-17) broke the build on macOS with sha1dc by bypassing our
> hash abstraction (git_SHA_CTX etc.), but it wasn't caught before the
> problematic topic was merged down to the 'master' branch. Nobody
> was even compile testing with DC_SHA1 set, although it is the
> recommended choice in these days for folks when they use SHA-1.
>
> This was because the default for macOS uses Apple Common Crypto, and
> both of the two CI jobs did not override the default. Tweak one of
> them to use DC_SHA1 to improve the coverage.
Thanks. This revised commit message does a much better job of
explaining the problem the patch is addressing.
^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [PATCH v2 00/12] fsmonitor: Implement fsmonitor for Linux
2022-10-17 22:14 ` Glen Choo
2022-10-18 4:17 ` Junio C Hamano
@ 2022-10-20 15:43 ` Junio C Hamano
2022-10-20 22:01 ` Junio C Hamano
1 sibling, 1 reply; 89+ messages in thread
From: Junio C Hamano @ 2022-10-20 15:43 UTC (permalink / raw)
To: Glen Choo
Cc: Eric DeCosta via GitGitGadget, git, Eric Sunshine,
Ævar Arnfjörð Bjarmason, Eric DeCosta
Glen Choo <chooglen@google.com> writes:
> At $DAYJOB, we observed that this topic breaks MacOS builds with sha1dc:
>
> $ make NO_APPLE_COMMON_CRYPTO=1 DC_SHA1=1 NO_OPENSSL=1 compat/fsmonitor/fsm-ipc-darwin.o
Glen, Ævar has cherry-picked the SHA_CTX -> git_SHA_CTX build fix
and I have merged it to 'next'. Can you test-build to see if that
change is sufficient to fix "does not even build from the source"
issue for you guys? You do not have to go through any official full
qualification cycle or release procedure---just checking that you
would be OK when you need to do the build the next time before it
has to happen is sufficient.
Thanks.
^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [PATCH v2 00/12] fsmonitor: Implement fsmonitor for Linux
2022-10-20 15:43 ` Junio C Hamano
@ 2022-10-20 22:01 ` Junio C Hamano
0 siblings, 0 replies; 89+ messages in thread
From: Junio C Hamano @ 2022-10-20 22:01 UTC (permalink / raw)
To: Glen Choo
Cc: Eric DeCosta via GitGitGadget, git, Eric Sunshine,
Ævar Arnfjörð Bjarmason, Eric DeCosta
Junio C Hamano <gitster@pobox.com> writes:
> Glen Choo <chooglen@google.com> writes:
>
>> At $DAYJOB, we observed that this topic breaks MacOS builds with sha1dc:
>>
>> $ make NO_APPLE_COMMON_CRYPTO=1 DC_SHA1=1 NO_OPENSSL=1 compat/fsmonitor/fsm-ipc-darwin.o
>
> Glen, Ævar has cherry-picked the SHA_CTX -> git_SHA_CTX build fix
> and I have merged it to 'next'. Can you test-build to see if that
> change is sufficient to fix "does not even build from the source"
> issue for you guys? You do not have to go through any official full
> qualification cycle or release procedure---just checking that you
> would be OK when you need to do the build the next time before it
> has to happen is sufficient.
>
> Thanks.
FYI, https://github.com/git/git/actions/runs/3292710187 is a
successful (wow, what's a rare event these days) run of CI on 'seen'
that includes
* Peff's -O0 fix for unconfusing LSan to prevent it from giving
false positives
* Eric DeCosta's change, cherry-picked by Ævar, to let fsmonitor
stuff compile with sha1dc as well as Apple's Common Crypto.
* A tweak to make one of the macOS CI jobs to build with sha1dc.
among all the other topics in flight. I plan to merge all of them
to 'next' (I think the first two are already in 'next') and fast
track them down to 'master'.
^ permalink raw reply [flat|nested] 89+ messages in thread
* [PATCH v3 0/6] fsmonitor: Implement fsmonitor for Linux
2022-10-14 21:45 ` [PATCH v2 " Eric DeCosta via GitGitGadget
` (13 preceding siblings ...)
2022-10-17 22:14 ` Glen Choo
@ 2022-11-16 23:23 ` Eric DeCosta via GitGitGadget
2022-11-16 23:23 ` [PATCH v3 1/6] fsmonitor: prepare to share code between Mac OS and Linux Eric DeCosta via GitGitGadget
` (7 more replies)
14 siblings, 8 replies; 89+ messages in thread
From: Eric DeCosta via GitGitGadget @ 2022-11-16 23:23 UTC (permalink / raw)
To: git
Cc: Eric Sunshine, Ævar Arnfjörð Bjarmason, Glen Choo,
Johannes Schindelin, Eric DeCosta
Goal is to deliver fsmonitor for Linux that is on par with fsmonitor for
Windows and Mac OS.
This patch set builds upon previous work for done for Windows and Mac OS to
implement a fsmonitor back-end for Linux based on the Linux inotify API.
inotify differs significantly from the equivalent Windows and Mac OS APIs in
that a watch must be registered for every directory of interest (rather than
a singular watch at the root of the directory tree) and special care must be
taken to handle directory renames correctly.
More information about inotify:
https://man7.org/linux/man-pages/man7/inotify.7.html
v2 differs from v1:
* Prior work for Windows and Mac OS has been merged to master, reducing the
patch set from 12 to 6 patches
* Code review feedback
* Identified and resolved race condition revealed by CI test system, see
"Limitations and caveats" regarding monitoring of directory trees from
the man page, above
* Apologies for being away from this for so long, but my attention was
needed elsewhere
v1 differs from v0:
* Code review feedback
* Update how and which code can be shared between Mac OS and Linux
* Increase polling frequency to every 1ms (matches Mac OS)
* Updates to t7527 to improve test stability
Eric DeCosta (6):
fsmonitor: prepare to share code between Mac OS and Linux
fsmonitor: determine if filesystem is local or remote
fsmonitor: implement filesystem change listener for Linux
fsmonitor: enable fsmonitor for Linux
fsmonitor: test updates
fsmonitor: update doc for Linux
Documentation/config/fsmonitor--daemon.txt | 4 +-
Documentation/git-fsmonitor--daemon.txt | 24 +-
compat/fsmonitor/fsm-health-linux.c | 24 +
compat/fsmonitor/fsm-ipc-darwin.c | 53 +-
compat/fsmonitor/fsm-ipc-linux.c | 1 +
compat/fsmonitor/fsm-ipc-unix.c | 52 ++
compat/fsmonitor/fsm-listen-linux.c | 676 +++++++++++++++++++++
compat/fsmonitor/fsm-path-utils-linux.c | 169 ++++++
compat/fsmonitor/fsm-settings-darwin.c | 63 +-
compat/fsmonitor/fsm-settings-linux.c | 1 +
compat/fsmonitor/fsm-settings-unix.c | 61 ++
config.mak.uname | 8 +
contrib/buildsystems/CMakeLists.txt | 11 +-
t/t7527-builtin-fsmonitor.sh | 94 ++-
14 files changed, 1102 insertions(+), 139 deletions(-)
create mode 100644 compat/fsmonitor/fsm-health-linux.c
create mode 100644 compat/fsmonitor/fsm-ipc-linux.c
create mode 100644 compat/fsmonitor/fsm-ipc-unix.c
create mode 100644 compat/fsmonitor/fsm-listen-linux.c
create mode 100644 compat/fsmonitor/fsm-path-utils-linux.c
create mode 100644 compat/fsmonitor/fsm-settings-linux.c
create mode 100644 compat/fsmonitor/fsm-settings-unix.c
base-commit: 319605f8f00e402f3ea758a02c63534ff800a711
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1352%2Fedecosta-mw%2Ffsmonitor_linux-v3
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1352/edecosta-mw/fsmonitor_linux-v3
Pull-Request: https://github.com/git/git/pull/1352
Range-diff vs v2:
1: cd46bed37c3 < -: ----------- fsmonitor: refactor filesystem checks to common interface
2: 21d114bda4b < -: ----------- fsmonitor: relocate socket file if .git directory is remote
3: 664259ed57a < -: ----------- fsmonitor: avoid socket location check if using hook
4: d8f20032d6b < -: ----------- fsmonitor: deal with synthetic firmlinks on macOS
5: ab1e0ab967c < -: ----------- fsmonitor: check for compatability before communicating with fsmonitor
6: 9c552239b57 < -: ----------- fsmonitor: add documentation for allowRemote and socketDir options
7: 295beb89ab1 ! 1: 99d684c7bdf fsmonitor: prepare to share code between Mac OS and Linux
@@ Commit message
Linux and Mac OS can share some of the code originally developed for Mac OS.
- Minor update to compat/fsmonitor/fsm-ipc-unix.c to make it cross-platform.
- Mac OS and Linux can share fsm-ipc-unix.c
-
- Both platforms can also share compat/fsmonitor/fsm-settings-unix.c but we
- will leave room for future, platform-specific checks by having the platform-
- specific implementations call into fsm-settings-unix.
+ Mac OS and Linux can share fsm-ipc-unix.c and fsm-settings-unix.c
Signed-off-by: Eric DeCosta <edecosta@mathworks.com>
- ## Makefile ##
-@@ Makefile: endif
-
- ifdef FSMONITOR_DAEMON_BACKEND
- COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND
-+ ifdef FSMONITOR_DAEMON_COMMON
-+ COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_DAEMON_COMMON).o
-+ else
-+ COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_DAEMON_BACKEND).o
-+ endif
- COMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o
- COMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_BACKEND).o
-- COMPAT_OBJS += compat/fsmonitor/fsm-ipc-$(FSMONITOR_DAEMON_BACKEND).o
- endif
-
- ifdef FSMONITOR_OS_SETTINGS
- COMPAT_CFLAGS += -DHAVE_FSMONITOR_OS_SETTINGS
-+ ifdef FSMONITOR_DAEMON_COMMON
-+ COMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_DAEMON_COMMON).o
-+ endif
- COMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_OS_SETTINGS).o
- COMPAT_OBJS += compat/fsmonitor/fsm-path-utils-$(FSMONITOR_OS_SETTINGS).o
- endif
-
## compat/fsmonitor/fsm-health-linux.c (new) ##
@@
+#include "cache.h"
@@ compat/fsmonitor/fsm-health-linux.c (new)
+{
+}
- ## compat/fsmonitor/fsm-ipc-darwin.c => compat/fsmonitor/fsm-ipc-unix.c ##
-@@ compat/fsmonitor/fsm-ipc-unix.c: static GIT_PATH_FUNC(fsmonitor_ipc__get_default_path, "fsmonitor--daemon.ipc")
- const char *fsmonitor_ipc__get_path(struct repository *r)
- {
- static const char *ipc_path = NULL;
-- SHA_CTX sha1ctx;
+ ## compat/fsmonitor/fsm-ipc-darwin.c ##
+@@
+-#include "cache.h"
+-#include "config.h"
+-#include "strbuf.h"
+-#include "fsmonitor.h"
+-#include "fsmonitor-ipc.h"
+-#include "fsmonitor-path-utils.h"
+-
+-static GIT_PATH_FUNC(fsmonitor_ipc__get_default_path, "fsmonitor--daemon.ipc")
+-
+-const char *fsmonitor_ipc__get_path(struct repository *r)
+-{
+- static const char *ipc_path = NULL;
+- git_SHA_CTX sha1ctx;
+- char *sock_dir = NULL;
+- struct strbuf ipc_file = STRBUF_INIT;
+- unsigned char hash[GIT_MAX_RAWSZ];
+-
+- if (!r)
+- BUG("No repository passed into fsmonitor_ipc__get_path");
+-
+- if (ipc_path)
+- return ipc_path;
+-
+-
+- /* By default the socket file is created in the .git directory */
+- if (fsmonitor__is_fs_remote(r->gitdir) < 1) {
+- ipc_path = fsmonitor_ipc__get_default_path();
+- return ipc_path;
+- }
+-
+- git_SHA1_Init(&sha1ctx);
+- git_SHA1_Update(&sha1ctx, r->worktree, strlen(r->worktree));
+- git_SHA1_Final(hash, &sha1ctx);
+-
+- repo_config_get_string(r, "fsmonitor.socketdir", &sock_dir);
+-
+- /* Create the socket file in either socketDir or $HOME */
+- if (sock_dir && *sock_dir) {
+- strbuf_addf(&ipc_file, "%s/.git-fsmonitor-%s",
+- sock_dir, hash_to_hex(hash));
+- } else {
+- strbuf_addf(&ipc_file, "~/.git-fsmonitor-%s", hash_to_hex(hash));
+- }
+- free(sock_dir);
+-
+- ipc_path = interpolate_path(ipc_file.buf, 1);
+- if (!ipc_path)
+- die(_("Invalid path: %s"), ipc_file.buf);
+-
+- strbuf_release(&ipc_file);
+- return ipc_path;
+-}
++#include "fsm-ipc-unix.c"
+
+ ## compat/fsmonitor/fsm-ipc-linux.c (new) ##
+@@
++#include "fsm-ipc-unix.c"
+
+ ## compat/fsmonitor/fsm-ipc-unix.c (new) ##
+@@
++#include "cache.h"
++#include "config.h"
++#include "strbuf.h"
++#include "fsmonitor.h"
++#include "fsmonitor-ipc.h"
++#include "fsmonitor-path-utils.h"
++
++static GIT_PATH_FUNC(fsmonitor_ipc__get_default_path, "fsmonitor--daemon.ipc")
++
++const char *fsmonitor_ipc__get_path(struct repository *r)
++{
++ static const char *ipc_path = NULL;
+ git_SHA_CTX sha1ctx;
- char *sock_dir = NULL;
- struct strbuf ipc_file = STRBUF_INIT;
- unsigned char hash[SHA_DIGEST_LENGTH];
-@@ compat/fsmonitor/fsm-ipc-unix.c: const char *fsmonitor_ipc__get_path(struct repository *r)
- return ipc_path;
- }
-
-- SHA1_Init(&sha1ctx);
-- SHA1_Update(&sha1ctx, r->worktree, strlen(r->worktree));
-- SHA1_Final(hash, &sha1ctx);
++ char *sock_dir = NULL;
++ struct strbuf ipc_file = STRBUF_INIT;
++ unsigned char hash[GIT_MAX_RAWSZ];
++
++ if (!r)
++ BUG("No repository passed into fsmonitor_ipc__get_path");
++
++ if (ipc_path)
++ return ipc_path;
++
++
++ /* By default the socket file is created in the .git directory */
++ if (fsmonitor__is_fs_remote(r->gitdir) < 1) {
++ ipc_path = fsmonitor_ipc__get_default_path();
++ return ipc_path;
++ }
++
+ git_SHA1_Init(&sha1ctx);
+ git_SHA1_Update(&sha1ctx, r->worktree, strlen(r->worktree));
+ git_SHA1_Final(hash, &sha1ctx);
-
- repo_config_get_string(r, "fsmonitor.socketdir", &sock_dir);
-
++
++ repo_config_get_string(r, "fsmonitor.socketdir", &sock_dir);
++
++ /* Create the socket file in either socketDir or $HOME */
++ if (sock_dir && *sock_dir) {
++ strbuf_addf(&ipc_file, "%s/.git-fsmonitor-%s",
++ sock_dir, hash_to_hex(hash));
++ } else {
++ strbuf_addf(&ipc_file, "~/.git-fsmonitor-%s", hash_to_hex(hash));
++ }
++ free(sock_dir);
++
++ ipc_path = interpolate_path(ipc_file.buf, 1);
++ if (!ipc_path)
++ die(_("Invalid path: %s"), ipc_file.buf);
++
++ strbuf_release(&ipc_file);
++ return ipc_path;
++}
## compat/fsmonitor/fsm-settings-darwin.c ##
@@
- #include "config.h"
- #include "fsmonitor.h"
- #include "fsmonitor-ipc.h"
+-#include "config.h"
+-#include "fsmonitor.h"
+-#include "fsmonitor-ipc.h"
-#include "fsmonitor-settings.h"
- #include "fsmonitor-path-utils.h"
+-#include "fsmonitor-path-utils.h"
-
- /*
- * For the builtin FSMonitor, we create the Unix domain socket for the
@@ compat/fsmonitor/fsm-settings-darwin.c
- free(fs.typename);
- return FSMONITOR_REASON_OK;
-}
-+#include "fsmonitor-settings.h"
-+#include "fsm-settings-unix.h"
-
- enum fsmonitor_reason fsm_os__incompatible(struct repository *r, int ipc)
- {
+-
+-enum fsmonitor_reason fsm_os__incompatible(struct repository *r, int ipc)
+-{
- enum fsmonitor_reason reason;
-
- if (ipc) {
@@ compat/fsmonitor/fsm-settings-darwin.c
- }
-
- return FSMONITOR_REASON_OK;
-+ return fsm_os__incompatible_unix(r, ipc);
- }
+-}
++#include "fsm-settings-unix.c"
## compat/fsmonitor/fsm-settings-linux.c (new) ##
@@
-+#include "config.h"
-+#include "fsmonitor.h"
-+#include "fsmonitor-ipc.h"
-+#include "fsmonitor-path-utils.h"
-+#include "fsmonitor-settings.h"
-+#include "fsm-settings-unix.h"
-+
-+enum fsmonitor_reason fsm_os__incompatible(struct repository *r, int ipc)
-+{
-+ return fsm_os__incompatible_unix(r, ipc);
-+}
++#include "fsm-settings-unix.c"
## compat/fsmonitor/fsm-settings-unix.c (new) ##
@@
@@ compat/fsmonitor/fsm-settings-unix.c (new)
+#include "fsmonitor.h"
+#include "fsmonitor-ipc.h"
+#include "fsmonitor-path-utils.h"
-+#include "fsm-settings-unix.h"
+
+ /*
+ * For the builtin FSMonitor, we create the Unix domain socket for the
@@ compat/fsmonitor/fsm-settings-unix.c (new)
+ struct fs_info fs;
+ const char *ipc_path = fsmonitor_ipc__get_path(r);
+ struct strbuf path = STRBUF_INIT;
-+ strbuf_add(&path, ipc_path, strlen(ipc_path));
++ strbuf_addstr(&path, ipc_path);
+
+ if (fsmonitor__get_fs_info(dirname(path.buf), &fs) == -1) {
+ strbuf_release(&path);
@@ compat/fsmonitor/fsm-settings-unix.c (new)
+ return FSMONITOR_REASON_OK;
+}
+
-+enum fsmonitor_reason fsm_os__incompatible_unix(struct repository *r, int ipc)
++enum fsmonitor_reason fsm_os__incompatible(struct repository *r, int ipc)
+{
+ enum fsmonitor_reason reason;
+
@@ compat/fsmonitor/fsm-settings-unix.c (new)
+
+ return FSMONITOR_REASON_OK;
+}
-
- ## compat/fsmonitor/fsm-settings-unix.h (new) ##
-@@
-+#ifndef FSM_SETTINGS_UNIX_H
-+#define FSM_SETTINGS_UNIX_H
-+
-+#ifdef HAVE_FSMONITOR_OS_SETTINGS
-+/*
-+ * Check for compatibility on unix-like systems (e.g. Darwin and Linux)
-+ */
-+enum fsmonitor_reason fsm_os__incompatible_unix(struct repository *r, int ipc);
-+#endif /* HAVE_FSMONITOR_OS_SETTINGS */
-+
-+#endif /* FSM_SETTINGS_UNIX_H */
-
- ## config.mak.uname ##
-@@ config.mak.uname: ifeq ($(uname_S),Darwin)
- ifndef NO_UNIX_SOCKETS
- FSMONITOR_DAEMON_BACKEND = darwin
- FSMONITOR_OS_SETTINGS = darwin
-+ FSMONITOR_DAEMON_COMMON = unix
- endif
- endif
-
-
- ## contrib/buildsystems/CMakeLists.txt ##
-@@ contrib/buildsystems/CMakeLists.txt: else()
- endif()
-
- if(SUPPORTS_SIMPLE_IPC)
-- if(CMAKE_SYSTEM_NAME STREQUAL "Windows")
-+ if(CMAKE_SYSTEM_NAME STREQUAL "Linux")
-+ add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)
-+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-unix.c)
-+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-linux.c)
-+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-linux.c)
-+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-linux.c)
-+
-+ add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)
-+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-unix.c)
-+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-linux.c)
-+ elseif(CMAKE_SYSTEM_NAME STREQUAL "Windows")
- add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)
- list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)
- list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-win32.c)
-@@ contrib/buildsystems/CMakeLists.txt: if(SUPPORTS_SIMPLE_IPC)
- list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c)
- elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin")
- add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)
-- list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)
-+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-unix.c)
- list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-darwin.c)
-- list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-darwin.c)
-+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c)
- list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-darwin.c)
-
- add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)
-+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-unix.c)
- list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c)
- endif()
- endif()
8: 7d7ef78728f = 2: aa405379891 fsmonitor: determine if filesystem is local or remote
9: 4f9c5358475 ! 3: c2e5a7201aa fsmonitor: implement filesystem change listener for Linux
@@ compat/fsmonitor/fsm-listen-linux.c (new)
+/*
+ * Recursively add watches to every directory under path
+ */
-+static int register_inotify(const char *path, struct fsm_listen_data *data)
++static int register_inotify(const char *path,
++ struct fsmonitor_daemon_state *state,
++ struct fsmonitor_batch *batch)
+{
+ DIR *dir;
++ const char *rel;
+ struct strbuf current = STRBUF_INIT;
+ struct dirent *de;
+ struct stat fs;
@@ compat/fsmonitor/fsm-listen-linux.c (new)
+
+ /* recurse into directory */
+ if (S_ISDIR(fs.st_mode)) {
-+ if (add_watch(current.buf, data))
++ if (add_watch(current.buf, state->listen_data))
+ goto failed;
-+ if (register_inotify(current.buf, data))
++ if (register_inotify(current.buf, state, batch))
+ goto failed;
++ } else if (batch) {
++ rel = current.buf + state->path_worktree_watch.len + 1;
++ trace_printf_key(&trace_fsmonitor, "explicitly adding '%s'", rel);
++ fsmonitor_batch__add_path(batch, rel);
+ }
+ }
+ ret = 0;
@@ compat/fsmonitor/fsm-listen-linux.c (new)
+
+ if (add_watch(state->path_worktree_watch.buf, data))
+ ret = -1;
-+ else if (register_inotify(state->path_worktree_watch.buf, data))
++ else if (register_inotify(state->path_worktree_watch.buf, state, NULL))
+ ret = -1;
+ else if (state->nr_paths_watching > 1) {
+ if (add_watch(state->path_gitdir_watch.buf, data))
+ ret = -1;
-+ else if (register_inotify(state->path_gitdir_watch.buf, data))
++ else if (register_inotify(state->path_gitdir_watch.buf, state, NULL))
+ ret = -1;
+ }
+
@@ compat/fsmonitor/fsm-listen-linux.c (new)
+ add_dir_rename(event->cookie, path, state->listen_data);
+
+ /* received IN_MOVE_TO, update watch to reflect new path */
-+ if (em_rename_dir_to(event->mask))
++ if (em_rename_dir_to(event->mask)) {
+ rename_dir(event->cookie, path, state->listen_data);
++ if (register_inotify(path, state, batch)) {
++ state->listen_data->shutdown = SHUTDOWN_ERROR;
++ goto done;
++ }
++ }
+
+ if (em_dir_created(event->mask)) {
+ if (add_watch(path, state->listen_data)) {
+ state->listen_data->shutdown = SHUTDOWN_ERROR;
+ goto done;
+ }
-+ if (register_inotify(path, state->listen_data)) {
++ if (register_inotify(path, state, batch)) {
+ state->listen_data->shutdown = SHUTDOWN_ERROR;
+ goto done;
+ }
@@ compat/fsmonitor/fsm-listen-linux.c (new)
+ strbuf_reset(&path);
+ strbuf_add(&path, w->dir, strlen(w->dir));
+ strbuf_addch(&path, '/');
-+ strbuf_add(&path, event->name, strlen(event->name));
++ strbuf_addstr(&path, event->name);
+
+ p = fsmonitor__resolve_alias(path.buf, &state->alias);
+ if (!p)
10: 07650ecd27b ! 4: 05f5b2dd4fb fsmonitor: enable fsmonitor for Linux
@@ config.mak.uname: ifeq ($(uname_S),Linux)
+ ifndef NO_UNIX_SOCKETS
+ FSMONITOR_DAEMON_BACKEND = linux
+ FSMONITOR_OS_SETTINGS = linux
-+ FSMONITOR_DAEMON_COMMON = unix
+ endif
+ endif
endif
ifeq ($(uname_S),GNU/kFreeBSD)
HAVE_ALLOCA_H = YesPlease
+
+ ## contrib/buildsystems/CMakeLists.txt ##
+@@ contrib/buildsystems/CMakeLists.txt: else()
+ endif()
+
+ if(SUPPORTS_SIMPLE_IPC)
+- if(CMAKE_SYSTEM_NAME STREQUAL "Windows")
++ if(CMAKE_SYSTEM_NAME STREQUAL "Linux")
++ add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)
++ list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-linux.c)
++ list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-linux.c)
++ list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-linux.c)
++ list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-linux.c)
++
++ add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)
++ list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-linux.c)
++ elseif(CMAKE_SYSTEM_NAME STREQUAL "Windows")
+ add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)
+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)
+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-win32.c)
11: 6682938fff8 ! 5: 754355ca44f fsmonitor: test updates
@@ t/t7527-builtin-fsmonitor.sh: start_daemon () {
+ then
+ if test "$last" -eq "$nsz"
+ then
++ cat "$file" &&
+ return 0
+ fi
+ last=$nsz
@@ t/t7527-builtin-fsmonitor.sh: start_daemon () {
+ sleep 1
+ k=$(( $k + 1 ))
+ done &&
++ cat "$file" &&
+ return 0
+}
+
@@ t/t7527-builtin-fsmonitor.sh: create_files () {
+rename_directory () {
+ mv dirtorename dirrenamed
+}
++
++rename_directory_file () {
++ mv dirtorename dirrenamed &&
++ echo 1 > dirrenamed/new
++}
+
rename_files () {
mv rename renamed &&
@@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'rename directory' '
grep "^event: dirtorename/*$" .git/trace &&
grep "^event: dirrenamed/*$" .git/trace
'
-@@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'file changes to directory' '
+
++test_expect_success 'rename directory file' '
++ test_when_finished clean_up_repo_and_stop_daemon &&
++
++ start_daemon --tf "$PWD/.git/trace" &&
++
++ wait_for_update rename_directory_file "$PWD/.git/trace" &&
++
++ test-tool fsmonitor-client query --token 0 &&
++
++ test_might_fail git fsmonitor--daemon stop &&
++
++ grep "^event: dirtorename/*$" .git/trace &&
++ grep "^event: dirrenamed/*$" .git/trace &&
++ grep "^event: dirrenamed/new$" .git/trace
++'
+ test_expect_success 'file changes to directory' '
+ test_when_finished clean_up_repo_and_stop_daemon &&
start_daemon --tf "$PWD/.git/trace" &&
@@ t/t7527-builtin-fsmonitor.sh: u_values="$u1 $u2"
git init "$u" &&
echo 1 >"$u"/file1 &&
-@@ t/t7527-builtin-fsmonitor.sh: my_match_and_clean () {
- }
+@@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'submodule setup' '
+ '
test_expect_success 'submodule always visited' '
- test_when_finished "git -C super fsmonitor--daemon stop; \
12: dd73e126810 = 6: f56175e097a fsmonitor: update doc for Linux
--
gitgitgadget
^ permalink raw reply [flat|nested] 89+ messages in thread* [PATCH v3 1/6] fsmonitor: prepare to share code between Mac OS and Linux
2022-11-16 23:23 ` [PATCH v3 0/6] " Eric DeCosta via GitGitGadget
@ 2022-11-16 23:23 ` Eric DeCosta via GitGitGadget
2022-11-16 23:23 ` [PATCH v3 2/6] fsmonitor: determine if filesystem is local or remote Eric DeCosta via GitGitGadget
` (6 subsequent siblings)
7 siblings, 0 replies; 89+ messages in thread
From: Eric DeCosta via GitGitGadget @ 2022-11-16 23:23 UTC (permalink / raw)
To: git
Cc: Eric Sunshine, Ævar Arnfjörð Bjarmason, Glen Choo,
Johannes Schindelin, Eric DeCosta, Eric DeCosta
From: Eric DeCosta <edecosta@mathworks.com>
Linux and Mac OS can share some of the code originally developed for Mac OS.
Mac OS and Linux can share fsm-ipc-unix.c and fsm-settings-unix.c
Signed-off-by: Eric DeCosta <edecosta@mathworks.com>
---
compat/fsmonitor/fsm-health-linux.c | 24 ++++++++++
compat/fsmonitor/fsm-ipc-darwin.c | 53 +---------------------
compat/fsmonitor/fsm-ipc-linux.c | 1 +
compat/fsmonitor/fsm-ipc-unix.c | 52 +++++++++++++++++++++
compat/fsmonitor/fsm-settings-darwin.c | 63 +-------------------------
compat/fsmonitor/fsm-settings-linux.c | 1 +
compat/fsmonitor/fsm-settings-unix.c | 61 +++++++++++++++++++++++++
7 files changed, 141 insertions(+), 114 deletions(-)
create mode 100644 compat/fsmonitor/fsm-health-linux.c
create mode 100644 compat/fsmonitor/fsm-ipc-linux.c
create mode 100644 compat/fsmonitor/fsm-ipc-unix.c
create mode 100644 compat/fsmonitor/fsm-settings-linux.c
create mode 100644 compat/fsmonitor/fsm-settings-unix.c
diff --git a/compat/fsmonitor/fsm-health-linux.c b/compat/fsmonitor/fsm-health-linux.c
new file mode 100644
index 00000000000..b9f709e8548
--- /dev/null
+++ b/compat/fsmonitor/fsm-health-linux.c
@@ -0,0 +1,24 @@
+#include "cache.h"
+#include "config.h"
+#include "fsmonitor.h"
+#include "fsm-health.h"
+#include "fsmonitor--daemon.h"
+
+int fsm_health__ctor(struct fsmonitor_daemon_state *state)
+{
+ return 0;
+}
+
+void fsm_health__dtor(struct fsmonitor_daemon_state *state)
+{
+ return;
+}
+
+void fsm_health__loop(struct fsmonitor_daemon_state *state)
+{
+ return;
+}
+
+void fsm_health__stop_async(struct fsmonitor_daemon_state *state)
+{
+}
diff --git a/compat/fsmonitor/fsm-ipc-darwin.c b/compat/fsmonitor/fsm-ipc-darwin.c
index d67b0ee50d3..4c3c92081ee 100644
--- a/compat/fsmonitor/fsm-ipc-darwin.c
+++ b/compat/fsmonitor/fsm-ipc-darwin.c
@@ -1,52 +1 @@
-#include "cache.h"
-#include "config.h"
-#include "strbuf.h"
-#include "fsmonitor.h"
-#include "fsmonitor-ipc.h"
-#include "fsmonitor-path-utils.h"
-
-static GIT_PATH_FUNC(fsmonitor_ipc__get_default_path, "fsmonitor--daemon.ipc")
-
-const char *fsmonitor_ipc__get_path(struct repository *r)
-{
- static const char *ipc_path = NULL;
- git_SHA_CTX sha1ctx;
- char *sock_dir = NULL;
- struct strbuf ipc_file = STRBUF_INIT;
- unsigned char hash[GIT_MAX_RAWSZ];
-
- if (!r)
- BUG("No repository passed into fsmonitor_ipc__get_path");
-
- if (ipc_path)
- return ipc_path;
-
-
- /* By default the socket file is created in the .git directory */
- if (fsmonitor__is_fs_remote(r->gitdir) < 1) {
- ipc_path = fsmonitor_ipc__get_default_path();
- return ipc_path;
- }
-
- git_SHA1_Init(&sha1ctx);
- git_SHA1_Update(&sha1ctx, r->worktree, strlen(r->worktree));
- git_SHA1_Final(hash, &sha1ctx);
-
- repo_config_get_string(r, "fsmonitor.socketdir", &sock_dir);
-
- /* Create the socket file in either socketDir or $HOME */
- if (sock_dir && *sock_dir) {
- strbuf_addf(&ipc_file, "%s/.git-fsmonitor-%s",
- sock_dir, hash_to_hex(hash));
- } else {
- strbuf_addf(&ipc_file, "~/.git-fsmonitor-%s", hash_to_hex(hash));
- }
- free(sock_dir);
-
- ipc_path = interpolate_path(ipc_file.buf, 1);
- if (!ipc_path)
- die(_("Invalid path: %s"), ipc_file.buf);
-
- strbuf_release(&ipc_file);
- return ipc_path;
-}
+#include "fsm-ipc-unix.c"
diff --git a/compat/fsmonitor/fsm-ipc-linux.c b/compat/fsmonitor/fsm-ipc-linux.c
new file mode 100644
index 00000000000..4c3c92081ee
--- /dev/null
+++ b/compat/fsmonitor/fsm-ipc-linux.c
@@ -0,0 +1 @@
+#include "fsm-ipc-unix.c"
diff --git a/compat/fsmonitor/fsm-ipc-unix.c b/compat/fsmonitor/fsm-ipc-unix.c
new file mode 100644
index 00000000000..d67b0ee50d3
--- /dev/null
+++ b/compat/fsmonitor/fsm-ipc-unix.c
@@ -0,0 +1,52 @@
+#include "cache.h"
+#include "config.h"
+#include "strbuf.h"
+#include "fsmonitor.h"
+#include "fsmonitor-ipc.h"
+#include "fsmonitor-path-utils.h"
+
+static GIT_PATH_FUNC(fsmonitor_ipc__get_default_path, "fsmonitor--daemon.ipc")
+
+const char *fsmonitor_ipc__get_path(struct repository *r)
+{
+ static const char *ipc_path = NULL;
+ git_SHA_CTX sha1ctx;
+ char *sock_dir = NULL;
+ struct strbuf ipc_file = STRBUF_INIT;
+ unsigned char hash[GIT_MAX_RAWSZ];
+
+ if (!r)
+ BUG("No repository passed into fsmonitor_ipc__get_path");
+
+ if (ipc_path)
+ return ipc_path;
+
+
+ /* By default the socket file is created in the .git directory */
+ if (fsmonitor__is_fs_remote(r->gitdir) < 1) {
+ ipc_path = fsmonitor_ipc__get_default_path();
+ return ipc_path;
+ }
+
+ git_SHA1_Init(&sha1ctx);
+ git_SHA1_Update(&sha1ctx, r->worktree, strlen(r->worktree));
+ git_SHA1_Final(hash, &sha1ctx);
+
+ repo_config_get_string(r, "fsmonitor.socketdir", &sock_dir);
+
+ /* Create the socket file in either socketDir or $HOME */
+ if (sock_dir && *sock_dir) {
+ strbuf_addf(&ipc_file, "%s/.git-fsmonitor-%s",
+ sock_dir, hash_to_hex(hash));
+ } else {
+ strbuf_addf(&ipc_file, "~/.git-fsmonitor-%s", hash_to_hex(hash));
+ }
+ free(sock_dir);
+
+ ipc_path = interpolate_path(ipc_file.buf, 1);
+ if (!ipc_path)
+ die(_("Invalid path: %s"), ipc_file.buf);
+
+ strbuf_release(&ipc_file);
+ return ipc_path;
+}
diff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c
index 6abbc7af3ab..14baf9f0603 100644
--- a/compat/fsmonitor/fsm-settings-darwin.c
+++ b/compat/fsmonitor/fsm-settings-darwin.c
@@ -1,62 +1 @@
-#include "config.h"
-#include "fsmonitor.h"
-#include "fsmonitor-ipc.h"
-#include "fsmonitor-settings.h"
-#include "fsmonitor-path-utils.h"
-
- /*
- * For the builtin FSMonitor, we create the Unix domain socket for the
- * IPC in the .git directory. If the working directory is remote,
- * then the socket will be created on the remote file system. This
- * can fail if the remote file system does not support UDS file types
- * (e.g. smbfs to a Windows server) or if the remote kernel does not
- * allow a non-local process to bind() the socket. (These problems
- * could be fixed by moving the UDS out of the .git directory and to a
- * well-known local directory on the client machine, but care should
- * be taken to ensure that $HOME is actually local and not a managed
- * file share.)
- *
- * FAT32 and NTFS working directories are problematic too.
- *
- * The builtin FSMonitor uses a Unix domain socket in the .git
- * directory for IPC. These Windows drive formats do not support
- * Unix domain sockets, so mark them as incompatible for the daemon.
- *
- */
-static enum fsmonitor_reason check_uds_volume(struct repository *r)
-{
- struct fs_info fs;
- const char *ipc_path = fsmonitor_ipc__get_path(r);
- struct strbuf path = STRBUF_INIT;
- strbuf_add(&path, ipc_path, strlen(ipc_path));
-
- if (fsmonitor__get_fs_info(dirname(path.buf), &fs) == -1) {
- strbuf_release(&path);
- return FSMONITOR_REASON_ERROR;
- }
-
- strbuf_release(&path);
-
- if (fs.is_remote ||
- !strcmp(fs.typename, "msdos") ||
- !strcmp(fs.typename, "ntfs")) {
- free(fs.typename);
- return FSMONITOR_REASON_NOSOCKETS;
- }
-
- free(fs.typename);
- return FSMONITOR_REASON_OK;
-}
-
-enum fsmonitor_reason fsm_os__incompatible(struct repository *r, int ipc)
-{
- enum fsmonitor_reason reason;
-
- if (ipc) {
- reason = check_uds_volume(r);
- if (reason != FSMONITOR_REASON_OK)
- return reason;
- }
-
- return FSMONITOR_REASON_OK;
-}
+#include "fsm-settings-unix.c"
diff --git a/compat/fsmonitor/fsm-settings-linux.c b/compat/fsmonitor/fsm-settings-linux.c
new file mode 100644
index 00000000000..14baf9f0603
--- /dev/null
+++ b/compat/fsmonitor/fsm-settings-linux.c
@@ -0,0 +1 @@
+#include "fsm-settings-unix.c"
diff --git a/compat/fsmonitor/fsm-settings-unix.c b/compat/fsmonitor/fsm-settings-unix.c
new file mode 100644
index 00000000000..a6ed32575a9
--- /dev/null
+++ b/compat/fsmonitor/fsm-settings-unix.c
@@ -0,0 +1,61 @@
+#include "config.h"
+#include "fsmonitor.h"
+#include "fsmonitor-ipc.h"
+#include "fsmonitor-path-utils.h"
+
+ /*
+ * For the builtin FSMonitor, we create the Unix domain socket for the
+ * IPC in the .git directory. If the working directory is remote,
+ * then the socket will be created on the remote file system. This
+ * can fail if the remote file system does not support UDS file types
+ * (e.g. smbfs to a Windows server) or if the remote kernel does not
+ * allow a non-local process to bind() the socket. (These problems
+ * could be fixed by moving the UDS out of the .git directory and to a
+ * well-known local directory on the client machine, but care should
+ * be taken to ensure that $HOME is actually local and not a managed
+ * file share.)
+ *
+ * FAT32 and NTFS working directories are problematic too.
+ *
+ * The builtin FSMonitor uses a Unix domain socket in the .git
+ * directory for IPC. These Windows drive formats do not support
+ * Unix domain sockets, so mark them as incompatible for the daemon.
+ *
+ */
+static enum fsmonitor_reason check_uds_volume(struct repository *r)
+{
+ struct fs_info fs;
+ const char *ipc_path = fsmonitor_ipc__get_path(r);
+ struct strbuf path = STRBUF_INIT;
+ strbuf_addstr(&path, ipc_path);
+
+ if (fsmonitor__get_fs_info(dirname(path.buf), &fs) == -1) {
+ strbuf_release(&path);
+ return FSMONITOR_REASON_ERROR;
+ }
+
+ strbuf_release(&path);
+
+ if (fs.is_remote ||
+ !strcmp(fs.typename, "msdos") ||
+ !strcmp(fs.typename, "ntfs")) {
+ free(fs.typename);
+ return FSMONITOR_REASON_NOSOCKETS;
+ }
+
+ free(fs.typename);
+ return FSMONITOR_REASON_OK;
+}
+
+enum fsmonitor_reason fsm_os__incompatible(struct repository *r, int ipc)
+{
+ enum fsmonitor_reason reason;
+
+ if (ipc) {
+ reason = check_uds_volume(r);
+ if (reason != FSMONITOR_REASON_OK)
+ return reason;
+ }
+
+ return FSMONITOR_REASON_OK;
+}
--
gitgitgadget
^ permalink raw reply related [flat|nested] 89+ messages in thread* [PATCH v3 2/6] fsmonitor: determine if filesystem is local or remote
2022-11-16 23:23 ` [PATCH v3 0/6] " Eric DeCosta via GitGitGadget
2022-11-16 23:23 ` [PATCH v3 1/6] fsmonitor: prepare to share code between Mac OS and Linux Eric DeCosta via GitGitGadget
@ 2022-11-16 23:23 ` Eric DeCosta via GitGitGadget
2022-11-16 23:23 ` [PATCH v3 3/6] fsmonitor: implement filesystem change listener for Linux Eric DeCosta via GitGitGadget
` (5 subsequent siblings)
7 siblings, 0 replies; 89+ messages in thread
From: Eric DeCosta via GitGitGadget @ 2022-11-16 23:23 UTC (permalink / raw)
To: git
Cc: Eric Sunshine, Ævar Arnfjörð Bjarmason, Glen Choo,
Johannes Schindelin, Eric DeCosta, Eric DeCosta
From: Eric DeCosta <edecosta@mathworks.com>
Compare the given path to the mounted filesystems. Find the mount that is
the longest prefix of the path (if any) and determine if that mount is on a
local or remote filesystem.
Signed-off-by: Eric DeCosta <edecosta@mathworks.com>
---
compat/fsmonitor/fsm-path-utils-linux.c | 169 ++++++++++++++++++++++++
1 file changed, 169 insertions(+)
create mode 100644 compat/fsmonitor/fsm-path-utils-linux.c
diff --git a/compat/fsmonitor/fsm-path-utils-linux.c b/compat/fsmonitor/fsm-path-utils-linux.c
new file mode 100644
index 00000000000..039f044b670
--- /dev/null
+++ b/compat/fsmonitor/fsm-path-utils-linux.c
@@ -0,0 +1,169 @@
+#include "fsmonitor.h"
+#include "fsmonitor-path-utils.h"
+#include <errno.h>
+#include <mntent.h>
+#include <sys/mount.h>
+#include <sys/statvfs.h>
+
+/*
+ * https://github.com/coreutils/gnulib/blob/master/lib/mountlist.c
+ */
+#ifndef ME_REMOTE
+/* A file system is "remote" if its Fs_name contains a ':'
+ or if (it is of type (smbfs or cifs) and its Fs_name starts with '//')
+ or if it is of any other of the listed types
+ or Fs_name is equal to "-hosts" (used by autofs to mount remote fs).
+ "VM" file systems like prl_fs or vboxsf are not considered remote here. */
+# define ME_REMOTE(Fs_name, Fs_type) \
+ (strchr (Fs_name, ':') != NULL \
+ || ((Fs_name)[0] == '/' \
+ && (Fs_name)[1] == '/' \
+ && (strcmp (Fs_type, "smbfs") == 0 \
+ || strcmp (Fs_type, "smb3") == 0 \
+ || strcmp (Fs_type, "cifs") == 0)) \
+ || strcmp (Fs_type, "acfs") == 0 \
+ || strcmp (Fs_type, "afs") == 0 \
+ || strcmp (Fs_type, "coda") == 0 \
+ || strcmp (Fs_type, "auristorfs") == 0 \
+ || strcmp (Fs_type, "fhgfs") == 0 \
+ || strcmp (Fs_type, "gpfs") == 0 \
+ || strcmp (Fs_type, "ibrix") == 0 \
+ || strcmp (Fs_type, "ocfs2") == 0 \
+ || strcmp (Fs_type, "vxfs") == 0 \
+ || strcmp ("-hosts", Fs_name) == 0)
+#endif
+
+static int find_mount(const char *path, const struct statvfs *fs,
+ struct mntent *ent)
+{
+ const char *const mounts = "/proc/mounts";
+ const char *rp = real_pathdup(path, 1);
+ struct mntent *ment = NULL;
+ struct statvfs mntfs;
+ FILE *fp;
+ int found = 0;
+ int dlen, plen, flen = 0;
+
+ ent->mnt_fsname = NULL;
+ ent->mnt_dir = NULL;
+ ent->mnt_type = NULL;
+
+ fp = setmntent(mounts, "r");
+ if (!fp) {
+ error_errno(_("setmntent('%s') failed"), mounts);
+ return -1;
+ }
+
+ plen = strlen(rp);
+
+ /* read all the mount information and compare to path */
+ while ((ment = getmntent(fp)) != NULL) {
+ if (statvfs(ment->mnt_dir, &mntfs)) {
+ switch (errno) {
+ case EPERM:
+ case ESRCH:
+ case EACCES:
+ continue;
+ default:
+ error_errno(_("statvfs('%s') failed"), ment->mnt_dir);
+ endmntent(fp);
+ return -1;
+ }
+ }
+
+ /* is mount on the same filesystem and is a prefix of the path */
+ if ((fs->f_fsid == mntfs.f_fsid) &&
+ !strncmp(ment->mnt_dir, rp, strlen(ment->mnt_dir))) {
+ dlen = strlen(ment->mnt_dir);
+ if (dlen > plen)
+ continue;
+ /*
+ * root is always a potential match; otherwise look for
+ * directory prefix
+ */
+ if ((dlen == 1 && ment->mnt_dir[0] == '/') ||
+ (dlen > flen && (!rp[dlen] || rp[dlen] == '/'))) {
+ flen = dlen;
+ /*
+ * https://man7.org/linux/man-pages/man3/getmntent.3.html
+ *
+ * The pointer points to a static area of memory which is
+ * overwritten by subsequent calls to getmntent().
+ */
+ found = 1;
+ free(ent->mnt_fsname);
+ free(ent->mnt_dir);
+ free(ent->mnt_type);
+ ent->mnt_fsname = xstrdup(ment->mnt_fsname);
+ ent->mnt_dir = xstrdup(ment->mnt_dir);
+ ent->mnt_type = xstrdup(ment->mnt_type);
+ }
+ }
+ }
+ endmntent(fp);
+
+ if (!found)
+ return -1;
+
+ return 0;
+}
+
+int fsmonitor__get_fs_info(const char *path, struct fs_info *fs_info)
+{
+ struct mntent ment;
+ struct statvfs fs;
+
+ if (statvfs(path, &fs))
+ return error_errno(_("statvfs('%s') failed"), path);
+
+
+ if (find_mount(path, &fs, &ment) < 0) {
+ free(ment.mnt_fsname);
+ free(ment.mnt_dir);
+ free(ment.mnt_type);
+ return -1;
+ }
+
+ trace_printf_key(&trace_fsmonitor,
+ "statvfs('%s') [flags 0x%08lx] '%s' '%s'",
+ path, fs.f_flag, ment.mnt_type, ment.mnt_fsname);
+
+ fs_info->is_remote = ME_REMOTE(ment.mnt_fsname, ment.mnt_type);
+ fs_info->typename = ment.mnt_fsname;
+ free(ment.mnt_dir);
+ free(ment.mnt_type);
+
+ trace_printf_key(&trace_fsmonitor,
+ "'%s' is_remote: %d",
+ path, fs_info->is_remote);
+ return 0;
+}
+
+int fsmonitor__is_fs_remote(const char *path)
+{
+ struct fs_info fs;
+
+ if (fsmonitor__get_fs_info(path, &fs))
+ return -1;
+
+ free(fs.typename);
+
+ return fs.is_remote;
+}
+
+/*
+ * No-op for now.
+ */
+int fsmonitor__get_alias(const char *path, struct alias_info *info)
+{
+ return 0;
+}
+
+/*
+ * No-op for now.
+ */
+char *fsmonitor__resolve_alias(const char *path,
+ const struct alias_info *info)
+{
+ return NULL;
+}
--
gitgitgadget
^ permalink raw reply related [flat|nested] 89+ messages in thread* [PATCH v3 3/6] fsmonitor: implement filesystem change listener for Linux
2022-11-16 23:23 ` [PATCH v3 0/6] " Eric DeCosta via GitGitGadget
2022-11-16 23:23 ` [PATCH v3 1/6] fsmonitor: prepare to share code between Mac OS and Linux Eric DeCosta via GitGitGadget
2022-11-16 23:23 ` [PATCH v3 2/6] fsmonitor: determine if filesystem is local or remote Eric DeCosta via GitGitGadget
@ 2022-11-16 23:23 ` Eric DeCosta via GitGitGadget
2022-11-16 23:23 ` [PATCH v3 4/6] fsmonitor: enable fsmonitor " Eric DeCosta via GitGitGadget
` (4 subsequent siblings)
7 siblings, 0 replies; 89+ messages in thread
From: Eric DeCosta via GitGitGadget @ 2022-11-16 23:23 UTC (permalink / raw)
To: git
Cc: Eric Sunshine, Ævar Arnfjörð Bjarmason, Glen Choo,
Johannes Schindelin, Eric DeCosta, Eric DeCosta
From: Eric DeCosta <edecosta@mathworks.com>
Implement a filesystem change listener for Linux based on the inotify API:
https://man7.org/linux/man-pages/man7/inotify.7.html
inotify requires registering a watch on every directory in the worktree and
special handling of moves/renames.
Signed-off-by: Eric DeCosta <edecosta@mathworks.com>
---
compat/fsmonitor/fsm-listen-linux.c | 676 ++++++++++++++++++++++++++++
1 file changed, 676 insertions(+)
create mode 100644 compat/fsmonitor/fsm-listen-linux.c
diff --git a/compat/fsmonitor/fsm-listen-linux.c b/compat/fsmonitor/fsm-listen-linux.c
new file mode 100644
index 00000000000..e8548e4e009
--- /dev/null
+++ b/compat/fsmonitor/fsm-listen-linux.c
@@ -0,0 +1,676 @@
+#include "cache.h"
+#include "fsmonitor.h"
+#include "fsm-listen.h"
+#include "fsmonitor--daemon.h"
+#include <dirent.h>
+#include <fcntl.h>
+#include <sys/inotify.h>
+#include <sys/stat.h>
+
+/*
+ * Safe value to bitwise OR with rest of mask for
+ * kernels that do not support IN_MASK_CREATE
+ */
+#ifndef IN_MASK_CREATE
+#define IN_MASK_CREATE 0x00000000
+#endif
+
+enum shutdown_reason {
+ SHUTDOWN_CONTINUE = 0,
+ SHUTDOWN_STOP,
+ SHUTDOWN_ERROR,
+ SHUTDOWN_FORCE
+};
+
+struct watch_entry {
+ struct hashmap_entry ent;
+ int wd;
+ uint32_t cookie;
+ const char *dir;
+};
+
+struct rename_entry {
+ struct hashmap_entry ent;
+ time_t whence;
+ uint32_t cookie;
+ const char *dir;
+};
+
+struct fsm_listen_data {
+ int fd_inotify;
+ enum shutdown_reason shutdown;
+ struct hashmap watches;
+ struct hashmap renames;
+ struct hashmap revwatches;
+};
+
+static int watch_entry_cmp(const void *cmp_data,
+ const struct hashmap_entry *eptr,
+ const struct hashmap_entry *entry_or_key,
+ const void *keydata)
+{
+ const struct watch_entry *e1, *e2;
+
+ e1 = container_of(eptr, const struct watch_entry, ent);
+ e2 = container_of(eptr, const struct watch_entry, ent);
+ return e1->wd != e2->wd;
+}
+
+static int revwatches_entry_cmp(const void *cmp_data,
+ const struct hashmap_entry *eptr,
+ const struct hashmap_entry *entry_or_key,
+ const void *keydata)
+{
+ const struct watch_entry *e1, *e2;
+
+ e1 = container_of(eptr, const struct watch_entry, ent);
+ e2 = container_of(eptr, const struct watch_entry, ent);
+ return strcmp(e1->dir, e2->dir);
+}
+
+static int rename_entry_cmp(const void *cmp_data,
+ const struct hashmap_entry *eptr,
+ const struct hashmap_entry *entry_or_key,
+ const void *keydata)
+{
+ const struct rename_entry *e1, *e2;
+
+ e1 = container_of(eptr, const struct rename_entry, ent);
+ e2 = container_of(eptr, const struct rename_entry, ent);
+ return e1->cookie != e2->cookie;
+}
+
+/*
+ * Register an inotify watch, add watch descriptor to path mapping
+ * and the reverse mapping.
+ */
+static int add_watch(const char *path, struct fsm_listen_data *data)
+{
+ const char *interned = strintern(path);
+ struct watch_entry *w1, *w2;
+
+ /* add the inotify watch, don't allow watches to be modified */
+ int wd = inotify_add_watch(data->fd_inotify, interned,
+ (IN_ALL_EVENTS | IN_ONLYDIR | IN_MASK_CREATE)
+ ^ IN_ACCESS ^ IN_CLOSE ^ IN_OPEN);
+ if (wd < 0)
+ return error_errno("inotify_add_watch('%s') failed", interned);
+
+ /* add watch descriptor -> directory mapping */
+ CALLOC_ARRAY(w1, 1);
+ w1->wd = wd;
+ w1->dir = interned;
+ hashmap_entry_init(&w1->ent, memhash(&w1->wd, sizeof(int)));
+ hashmap_add(&data->watches, &w1->ent);
+
+ /* add directory -> watch descriptor mapping */
+ CALLOC_ARRAY(w2, 1);
+ w2->wd = wd;
+ w2->dir = interned;
+ hashmap_entry_init(&w2->ent, memhash(w2->dir, strlen(w2->dir)));
+ hashmap_add(&data->revwatches, &w2->ent);
+
+ return 0;
+}
+
+/*
+ * Remove the inotify watch, the watch descriptor to path mapping
+ * and the reverse mapping.
+ */
+static void remove_watch(struct watch_entry *w,
+ struct fsm_listen_data *data)
+{
+ struct watch_entry k1, k2, *w1, *w2;
+
+ /* remove watch, ignore error if kernel already did it */
+ if (inotify_rm_watch(data->fd_inotify, w->wd) && errno != EINVAL)
+ error_errno("inotify_rm_watch() failed");
+
+ hashmap_entry_init(&k1.ent, memhash(&w->wd, sizeof(int)));
+ w1 = hashmap_remove_entry(&data->watches, &k1, ent, NULL);
+ if (!w1)
+ BUG("Double remove of watch for '%s'", w->dir);
+
+ if (w1->cookie)
+ BUG("Removing watch for '%s' which has a pending rename", w1->dir);
+
+ hashmap_entry_init(&k2.ent, memhash(w->dir, strlen(w->dir)));
+ w2 = hashmap_remove_entry(&data->revwatches, &k2, ent, NULL);
+ if (!w2)
+ BUG("Double remove of reverse watch for '%s'", w->dir);
+
+ /* w1->dir and w2->dir are interned strings, we don't own them */
+ free(w1);
+ free(w2);
+}
+
+/*
+ * Check for stale directory renames.
+ *
+ * https://man7.org/linux/man-pages/man7/inotify.7.html
+ *
+ * Allow for some small timeout to account for the fact that insertion of the
+ * IN_MOVED_FROM+IN_MOVED_TO event pair is not atomic, and the possibility that
+ * there may not be any IN_MOVED_TO event.
+ *
+ * If the IN_MOVED_TO event is not received within the timeout then events have
+ * been missed and the monitor is in an inconsistent state with respect to the
+ * filesystem.
+ */
+static int check_stale_dir_renames(struct hashmap *renames, time_t max_age)
+{
+ struct rename_entry *re;
+ struct hashmap_iter iter;
+
+ hashmap_for_each_entry(renames, &iter, re, ent) {
+ if (re->whence <= max_age)
+ return -1;
+ }
+ return 0;
+}
+
+/*
+ * Track pending renames.
+ *
+ * Tracking is done via a event cookie to watch descriptor mapping.
+ *
+ * A rename is not complete until matching a IN_MOVED_TO event is received
+ * for a corresponding IN_MOVED_FROM event.
+ */
+static void add_dir_rename(uint32_t cookie, const char *path,
+ struct fsm_listen_data *data)
+{
+ struct watch_entry k, *w;
+ struct rename_entry *re;
+
+ /* lookup the watch descriptor for the given path */
+ hashmap_entry_init(&k.ent, memhash(path, strlen(path)));
+ w = hashmap_get_entry(&data->revwatches, &k, ent, NULL);
+ if (!w) /* should never happen */
+ BUG("No watch for '%s'", path);
+ w->cookie = cookie;
+
+ /* add the pending rename to match against later */
+ CALLOC_ARRAY(re, 1);
+ re->dir = w->dir;
+ re->cookie = w->cookie;
+ re->whence = time(NULL);
+ hashmap_entry_init(&re->ent, memhash(&re->cookie, sizeof(uint32_t)));
+ hashmap_add(&data->renames, &re->ent);
+}
+
+/*
+ * Handle directory renames
+ *
+ * Once a IN_MOVED_TO event is received, lookup the rename tracking information
+ * via the event cookie and use this information to update the watch.
+ */
+static void rename_dir(uint32_t cookie, const char *path,
+ struct fsm_listen_data *data)
+{
+ struct rename_entry rek, *re;
+ struct watch_entry k, *w;
+
+ /* lookup a pending rename to match */
+ rek.cookie = cookie;
+ hashmap_entry_init(&rek.ent, memhash(&rek.cookie, sizeof(uint32_t)));
+ re = hashmap_get_entry(&data->renames, &rek, ent, NULL);
+ if (re) {
+ k.dir = re->dir;
+ hashmap_entry_init(&k.ent, memhash(k.dir, strlen(k.dir)));
+ w = hashmap_get_entry(&data->revwatches, &k, ent, NULL);
+ if (w) {
+ w->cookie = 0; /* rename handled */
+ remove_watch(w, data);
+ add_watch(path, data);
+ } else {
+ BUG("No matching watch");
+ }
+ } else {
+ BUG("No matching cookie");
+ }
+}
+
+/*
+ * Recursively add watches to every directory under path
+ */
+static int register_inotify(const char *path,
+ struct fsmonitor_daemon_state *state,
+ struct fsmonitor_batch *batch)
+{
+ DIR *dir;
+ const char *rel;
+ struct strbuf current = STRBUF_INIT;
+ struct dirent *de;
+ struct stat fs;
+ int ret = -1;
+
+ dir = opendir(path);
+ if (!dir)
+ return error_errno("opendir('%s') failed", path);
+
+ while ((de = readdir_skip_dot_and_dotdot(dir)) != NULL) {
+ strbuf_reset(¤t);
+ strbuf_addf(¤t, "%s/%s", path, de->d_name);
+ if (lstat(current.buf, &fs)) {
+ error_errno("lstat('%s') failed", current.buf);
+ goto failed;
+ }
+
+ /* recurse into directory */
+ if (S_ISDIR(fs.st_mode)) {
+ if (add_watch(current.buf, state->listen_data))
+ goto failed;
+ if (register_inotify(current.buf, state, batch))
+ goto failed;
+ } else if (batch) {
+ rel = current.buf + state->path_worktree_watch.len + 1;
+ trace_printf_key(&trace_fsmonitor, "explicitly adding '%s'", rel);
+ fsmonitor_batch__add_path(batch, rel);
+ }
+ }
+ ret = 0;
+
+failed:
+ strbuf_release(¤t);
+ if (closedir(dir) < 0)
+ return error_errno("closedir('%s') failed", path);
+ return ret;
+}
+
+static int em_rename_dir_from(u_int32_t mask)
+{
+ return ((mask & IN_ISDIR) && (mask & IN_MOVED_FROM));
+}
+
+static int em_rename_dir_to(u_int32_t mask)
+{
+ return ((mask & IN_ISDIR) && (mask & IN_MOVED_TO));
+}
+
+static int em_remove_watch(u_int32_t mask)
+{
+ return (mask & IN_DELETE_SELF);
+}
+
+static int em_dir_renamed(u_int32_t mask)
+{
+ return ((mask & IN_ISDIR) && (mask & IN_MOVE));
+}
+
+static int em_dir_created(u_int32_t mask)
+{
+ return ((mask & IN_ISDIR) && (mask & IN_CREATE));
+}
+
+static int em_dir_deleted(uint32_t mask)
+{
+ return ((mask & IN_ISDIR) && (mask & IN_DELETE));
+}
+
+static int em_force_shutdown(u_int32_t mask)
+{
+ return (mask & IN_UNMOUNT) || (mask & IN_Q_OVERFLOW);
+}
+
+static int em_ignore(u_int32_t mask)
+{
+ return (mask & IN_IGNORED) || (mask & IN_MOVE_SELF);
+}
+
+static void log_mask_set(const char *path, u_int32_t mask)
+{
+ struct strbuf msg = STRBUF_INIT;
+
+ if (mask & IN_ACCESS)
+ strbuf_addstr(&msg, "IN_ACCESS|");
+ if (mask & IN_MODIFY)
+ strbuf_addstr(&msg, "IN_MODIFY|");
+ if (mask & IN_ATTRIB)
+ strbuf_addstr(&msg, "IN_ATTRIB|");
+ if (mask & IN_CLOSE_WRITE)
+ strbuf_addstr(&msg, "IN_CLOSE_WRITE|");
+ if (mask & IN_CLOSE_NOWRITE)
+ strbuf_addstr(&msg, "IN_CLOSE_NOWRITE|");
+ if (mask & IN_OPEN)
+ strbuf_addstr(&msg, "IN_OPEN|");
+ if (mask & IN_MOVED_FROM)
+ strbuf_addstr(&msg, "IN_MOVED_FROM|");
+ if (mask & IN_MOVED_TO)
+ strbuf_addstr(&msg, "IN_MOVED_TO|");
+ if (mask & IN_CREATE)
+ strbuf_addstr(&msg, "IN_CREATE|");
+ if (mask & IN_DELETE)
+ strbuf_addstr(&msg, "IN_DELETE|");
+ if (mask & IN_DELETE_SELF)
+ strbuf_addstr(&msg, "IN_DELETE_SELF|");
+ if (mask & IN_MOVE_SELF)
+ strbuf_addstr(&msg, "IN_MOVE_SELF|");
+ if (mask & IN_UNMOUNT)
+ strbuf_addstr(&msg, "IN_UNMOUNT|");
+ if (mask & IN_Q_OVERFLOW)
+ strbuf_addstr(&msg, "IN_Q_OVERFLOW|");
+ if (mask & IN_IGNORED)
+ strbuf_addstr(&msg, "IN_IGNORED|");
+ if (mask & IN_ISDIR)
+ strbuf_addstr(&msg, "IN_ISDIR|");
+
+ trace_printf_key(&trace_fsmonitor, "inotify_event: '%s', mask=%#8.8x %s",
+ path, mask, msg.buf);
+
+ strbuf_release(&msg);
+}
+
+int fsm_listen__ctor(struct fsmonitor_daemon_state *state)
+{
+ int fd;
+ int ret = 0;
+ struct fsm_listen_data *data;
+
+ CALLOC_ARRAY(data, 1);
+ state->listen_data = data;
+ state->listen_error_code = -1;
+ data->shutdown = SHUTDOWN_ERROR;
+
+ fd = inotify_init1(O_NONBLOCK);
+ if (fd < 0)
+ return error_errno("inotify_init1() failed");
+
+ data->fd_inotify = fd;
+
+ hashmap_init(&data->watches, watch_entry_cmp, NULL, 0);
+ hashmap_init(&data->renames, rename_entry_cmp, NULL, 0);
+ hashmap_init(&data->revwatches, revwatches_entry_cmp, NULL, 0);
+
+ if (add_watch(state->path_worktree_watch.buf, data))
+ ret = -1;
+ else if (register_inotify(state->path_worktree_watch.buf, state, NULL))
+ ret = -1;
+ else if (state->nr_paths_watching > 1) {
+ if (add_watch(state->path_gitdir_watch.buf, data))
+ ret = -1;
+ else if (register_inotify(state->path_gitdir_watch.buf, state, NULL))
+ ret = -1;
+ }
+
+ if (!ret) {
+ state->listen_error_code = 0;
+ data->shutdown = SHUTDOWN_CONTINUE;
+ }
+
+ return ret;
+}
+
+void fsm_listen__dtor(struct fsmonitor_daemon_state *state)
+{
+ struct fsm_listen_data *data;
+ struct hashmap_iter iter;
+ struct watch_entry *w;
+ int fd;
+
+ if (!state || !state->listen_data)
+ return;
+
+ data = state->listen_data;
+ fd = data->fd_inotify;
+
+ hashmap_for_each_entry(&data->watches, &iter, w, ent) {
+ w->cookie = 0; /* ignore any pending renames */
+ remove_watch(w, data);
+ }
+ hashmap_clear(&data->watches);
+
+ hashmap_clear(&data->revwatches); /* remove_watch freed the entries */
+
+ hashmap_clear_and_free(&data->renames, struct rename_entry, ent);
+
+ FREE_AND_NULL(state->listen_data);
+
+ if (fd && (close(fd) < 0))
+ error_errno(_("closing inotify file descriptor failed"));
+}
+
+void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)
+{
+ if (!state->listen_data->shutdown)
+ state->listen_data->shutdown = SHUTDOWN_STOP;
+}
+
+/*
+ * Process a single inotify event and queue for publication.
+ */
+static int process_event(const char *path,
+ const struct inotify_event *event,
+ struct fsmonitor_batch *batch,
+ struct string_list *cookie_list,
+ struct fsmonitor_daemon_state *state)
+{
+ const char *rel;
+ const char *last_sep;
+
+ switch (fsmonitor_classify_path_absolute(state, path)) {
+ case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:
+ case IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:
+ /* Use just the filename of the cookie file. */
+ last_sep = find_last_dir_sep(path);
+ string_list_append(cookie_list,
+ last_sep ? last_sep + 1 : path);
+ break;
+ case IS_INSIDE_DOT_GIT:
+ case IS_INSIDE_GITDIR:
+ break;
+ case IS_DOT_GIT:
+ case IS_GITDIR:
+ /*
+ * If .git directory is deleted or renamed away,
+ * we have to quit.
+ */
+ if (em_dir_deleted(event->mask)) {
+ trace_printf_key(&trace_fsmonitor,
+ "event: gitdir removed");
+ state->listen_data->shutdown = SHUTDOWN_FORCE;
+ goto done;
+ }
+
+ if (em_dir_renamed(event->mask)) {
+ trace_printf_key(&trace_fsmonitor,
+ "event: gitdir renamed");
+ state->listen_data->shutdown = SHUTDOWN_FORCE;
+ goto done;
+ }
+ break;
+ case IS_WORKDIR_PATH:
+ /* normal events in the working directory */
+ if (trace_pass_fl(&trace_fsmonitor))
+ log_mask_set(path, event->mask);
+
+ rel = path + state->path_worktree_watch.len + 1;
+ fsmonitor_batch__add_path(batch, rel);
+
+ if (em_dir_deleted(event->mask))
+ break;
+
+ /* received IN_MOVE_FROM, add tracking for expected IN_MOVE_TO */
+ if (em_rename_dir_from(event->mask))
+ add_dir_rename(event->cookie, path, state->listen_data);
+
+ /* received IN_MOVE_TO, update watch to reflect new path */
+ if (em_rename_dir_to(event->mask)) {
+ rename_dir(event->cookie, path, state->listen_data);
+ if (register_inotify(path, state, batch)) {
+ state->listen_data->shutdown = SHUTDOWN_ERROR;
+ goto done;
+ }
+ }
+
+ if (em_dir_created(event->mask)) {
+ if (add_watch(path, state->listen_data)) {
+ state->listen_data->shutdown = SHUTDOWN_ERROR;
+ goto done;
+ }
+ if (register_inotify(path, state, batch)) {
+ state->listen_data->shutdown = SHUTDOWN_ERROR;
+ goto done;
+ }
+ }
+ break;
+ case IS_OUTSIDE_CONE:
+ default:
+ trace_printf_key(&trace_fsmonitor,
+ "ignoring '%s'", path);
+ break;
+ }
+ return 0;
+done:
+ return -1;
+}
+
+/*
+ * Read the inotify event stream and pre-process events before further
+ * processing and eventual publishing.
+ */
+static void handle_events(struct fsmonitor_daemon_state *state)
+{
+ /* See https://man7.org/linux/man-pages/man7/inotify.7.html */
+ char buf[4096]
+ __attribute__ ((aligned(__alignof__(struct inotify_event))));
+
+ struct hashmap watches = state->listen_data->watches;
+ struct fsmonitor_batch *batch = NULL;
+ struct string_list cookie_list = STRING_LIST_INIT_DUP;
+ struct watch_entry k, *w;
+ struct strbuf path;
+ const struct inotify_event *event;
+ int fd = state->listen_data->fd_inotify;
+ ssize_t len;
+ char *ptr, *p;
+
+ strbuf_init(&path, PATH_MAX);
+
+ for(;;) {
+ len = read(fd, buf, sizeof(buf));
+ if (len == -1 && errno != EAGAIN) {
+ error_errno(_("reading inotify message stream failed"));
+ state->listen_data->shutdown = SHUTDOWN_ERROR;
+ goto done;
+ }
+
+ /* nothing to read */
+ if (len <= 0)
+ goto done;
+
+ /* Loop over all events in the buffer. */
+ for (ptr = buf; ptr < buf + len;
+ ptr += sizeof(struct inotify_event) + event->len) {
+
+ event = (const struct inotify_event *) ptr;
+
+ if (em_ignore(event->mask))
+ continue;
+
+ /* File system was unmounted or event queue overflowed */
+ if (em_force_shutdown(event->mask)) {
+ if (trace_pass_fl(&trace_fsmonitor))
+ log_mask_set("Forcing shutdown", event->mask);
+ state->listen_data->shutdown = SHUTDOWN_FORCE;
+ goto done;
+ }
+
+ hashmap_entry_init(&k.ent, memhash(&event->wd, sizeof(int)));
+ k.wd = event->wd;
+
+ w = hashmap_get_entry(&watches, &k, ent, NULL);
+ if (!w) /* should never happen */
+ BUG("No watch for '%s'", event->name);
+
+ /* directory watch was removed */
+ if (em_remove_watch(event->mask)) {
+ remove_watch(w, state->listen_data);
+ continue;
+ }
+
+ strbuf_reset(&path);
+ strbuf_add(&path, w->dir, strlen(w->dir));
+ strbuf_addch(&path, '/');
+ strbuf_addstr(&path, event->name);
+
+ p = fsmonitor__resolve_alias(path.buf, &state->alias);
+ if (!p)
+ p = strbuf_detach(&path, NULL);
+
+ if (!batch)
+ batch = fsmonitor_batch__new();
+
+ if (process_event(p, event, batch, &cookie_list, state)) {
+ free(p);
+ goto done;
+ }
+ free(p);
+ }
+ strbuf_reset(&path);
+ fsmonitor_publish(state, batch, &cookie_list);
+ string_list_clear(&cookie_list, 0);
+ batch = NULL;
+ }
+done:
+ strbuf_release(&path);
+ fsmonitor_batch__free_list(batch);
+ string_list_clear(&cookie_list, 0);
+}
+
+/*
+ * Non-blocking read of the inotify events stream. The inotify fd is polled
+ * frequently to help minimize the number of queue overflows.
+ */
+void fsm_listen__loop(struct fsmonitor_daemon_state *state)
+{
+ int poll_num;
+ const int interval = 1000;
+ time_t checked = time(NULL);
+ struct pollfd fds[1];
+ fds[0].fd = state->listen_data->fd_inotify;
+ fds[0].events = POLLIN;
+
+ for(;;) {
+ switch (state->listen_data->shutdown) {
+ case SHUTDOWN_CONTINUE:
+ poll_num = poll(fds, 1, 1);
+ if (poll_num == -1) {
+ if (errno == EINTR)
+ continue;
+ error_errno(_("polling inotify message stream failed"));
+ state->listen_data->shutdown = SHUTDOWN_ERROR;
+ continue;
+ }
+
+ if ((time(NULL) - checked) >= interval) {
+ checked = time(NULL);
+ if (check_stale_dir_renames(&state->listen_data->renames,
+ checked - interval)) {
+ trace_printf_key(&trace_fsmonitor,
+ "Missed IN_MOVED_TO events, forcing shutdown");
+ state->listen_data->shutdown = SHUTDOWN_FORCE;
+ continue;
+ }
+ }
+
+ if (poll_num > 0 && (fds[0].revents & POLLIN))
+ handle_events(state);
+
+ continue;
+ case SHUTDOWN_ERROR:
+ state->listen_error_code = -1;
+ ipc_server_stop_async(state->ipc_server_data);
+ break;
+ case SHUTDOWN_FORCE:
+ state->listen_error_code = 0;
+ ipc_server_stop_async(state->ipc_server_data);
+ break;
+ case SHUTDOWN_STOP:
+ default:
+ state->listen_error_code = 0;
+ break;
+ }
+ return;
+ }
+}
--
gitgitgadget
^ permalink raw reply related [flat|nested] 89+ messages in thread* [PATCH v3 4/6] fsmonitor: enable fsmonitor for Linux
2022-11-16 23:23 ` [PATCH v3 0/6] " Eric DeCosta via GitGitGadget
` (2 preceding siblings ...)
2022-11-16 23:23 ` [PATCH v3 3/6] fsmonitor: implement filesystem change listener for Linux Eric DeCosta via GitGitGadget
@ 2022-11-16 23:23 ` Eric DeCosta via GitGitGadget
2022-11-16 23:23 ` [PATCH v3 5/6] fsmonitor: test updates Eric DeCosta via GitGitGadget
` (3 subsequent siblings)
7 siblings, 0 replies; 89+ messages in thread
From: Eric DeCosta via GitGitGadget @ 2022-11-16 23:23 UTC (permalink / raw)
To: git
Cc: Eric Sunshine, Ævar Arnfjörð Bjarmason, Glen Choo,
Johannes Schindelin, Eric DeCosta, Eric DeCosta
From: Eric DeCosta <edecosta@mathworks.com>
Update build to enable fsmonitor for Linux.
Signed-off-by: Eric DeCosta <edecosta@mathworks.com>
---
config.mak.uname | 8 ++++++++
contrib/buildsystems/CMakeLists.txt | 11 ++++++++++-
2 files changed, 18 insertions(+), 1 deletion(-)
diff --git a/config.mak.uname b/config.mak.uname
index d63629fe807..5890d810463 100644
--- a/config.mak.uname
+++ b/config.mak.uname
@@ -68,6 +68,14 @@ ifeq ($(uname_S),Linux)
ifneq ($(findstring .el7.,$(uname_R)),)
BASIC_CFLAGS += -std=c99
endif
+ # The builtin FSMonitor on Linux builds upon Simple-IPC. Both require
+ # Unix domain sockets and PThreads.
+ ifndef NO_PTHREADS
+ ifndef NO_UNIX_SOCKETS
+ FSMONITOR_DAEMON_BACKEND = linux
+ FSMONITOR_OS_SETTINGS = linux
+ endif
+ endif
endif
ifeq ($(uname_S),GNU/kFreeBSD)
HAVE_ALLOCA_H = YesPlease
diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt
index 3957e4cf8cd..f058c3d19b4 100644
--- a/contrib/buildsystems/CMakeLists.txt
+++ b/contrib/buildsystems/CMakeLists.txt
@@ -304,7 +304,16 @@ else()
endif()
if(SUPPORTS_SIMPLE_IPC)
- if(CMAKE_SYSTEM_NAME STREQUAL "Windows")
+ if(CMAKE_SYSTEM_NAME STREQUAL "Linux")
+ add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)
+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-linux.c)
+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-linux.c)
+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-linux.c)
+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-linux.c)
+
+ add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)
+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-linux.c)
+ elseif(CMAKE_SYSTEM_NAME STREQUAL "Windows")
add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)
list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)
list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-win32.c)
--
gitgitgadget
^ permalink raw reply related [flat|nested] 89+ messages in thread* [PATCH v3 5/6] fsmonitor: test updates
2022-11-16 23:23 ` [PATCH v3 0/6] " Eric DeCosta via GitGitGadget
` (3 preceding siblings ...)
2022-11-16 23:23 ` [PATCH v3 4/6] fsmonitor: enable fsmonitor " Eric DeCosta via GitGitGadget
@ 2022-11-16 23:23 ` Eric DeCosta via GitGitGadget
2022-11-16 23:23 ` [PATCH v3 6/6] fsmonitor: update doc for Linux Eric DeCosta via GitGitGadget
` (2 subsequent siblings)
7 siblings, 0 replies; 89+ messages in thread
From: Eric DeCosta via GitGitGadget @ 2022-11-16 23:23 UTC (permalink / raw)
To: git
Cc: Eric Sunshine, Ævar Arnfjörð Bjarmason, Glen Choo,
Johannes Schindelin, Eric DeCosta, Eric DeCosta
From: Eric DeCosta <edecosta@mathworks.com>
t7527-builtin-fsmonitor was leaking fsmonitor--daemon processes in some
cases.
Accomodate slight difference in the number of events generated on Linux.
On lower-powered systems, spin a little to give the daemon time
to respond to and log filesystem events.
Signed-off-by: Eric DeCosta <edecosta@mathworks.com>
---
t/t7527-builtin-fsmonitor.sh | 94 ++++++++++++++++++++++++++++++------
1 file changed, 80 insertions(+), 14 deletions(-)
diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh
index 4abc74db2bb..951374231b7 100755
--- a/t/t7527-builtin-fsmonitor.sh
+++ b/t/t7527-builtin-fsmonitor.sh
@@ -13,7 +13,7 @@ fi
stop_daemon_delete_repo () {
r=$1 &&
test_might_fail git -C $r fsmonitor--daemon stop &&
- rm -rf $1
+ rm -rf $r
}
start_daemon () {
@@ -72,6 +72,34 @@ start_daemon () {
)
}
+IMPLICIT_TIMEOUT=5
+
+wait_for_update () {
+ func=$1 &&
+ file=$2 &&
+ sz=$(wc -c < "$file") &&
+ last=0 &&
+ $func &&
+ k=0 &&
+ while test "$k" -lt $IMPLICIT_TIMEOUT
+ do
+ nsz=$(wc -c < "$file")
+ if test "$nsz" -gt "$sz"
+ then
+ if test "$last" -eq "$nsz"
+ then
+ cat "$file" &&
+ return 0
+ fi
+ last=$nsz
+ fi
+ sleep 1
+ k=$(( $k + 1 ))
+ done &&
+ cat "$file" &&
+ return 0
+}
+
# Is a Trace2 data event present with the given catetory and key?
# We do not care what the value is.
#
@@ -137,7 +165,6 @@ test_expect_success 'implicit daemon start' '
# machines (where it might take a moment to wake and reschedule the
# daemon process) to avoid false alarms during test runs.)
#
-IMPLICIT_TIMEOUT=5
verify_implicit_shutdown () {
r=$1 &&
@@ -373,6 +400,15 @@ create_files () {
echo 3 >dir2/new
}
+rename_directory () {
+ mv dirtorename dirrenamed
+}
+
+rename_directory_file () {
+ mv dirtorename dirrenamed &&
+ echo 1 > dirrenamed/new
+}
+
rename_files () {
mv rename renamed &&
mv dir1/rename dir1/renamed &&
@@ -427,10 +463,12 @@ test_expect_success 'edit some files' '
start_daemon --tf "$PWD/.git/trace" &&
- edit_files &&
+ wait_for_update edit_files "$PWD/.git/trace" &&
test-tool fsmonitor-client query --token 0 &&
+ test_might_fail git fsmonitor--daemon stop &&
+
grep "^event: dir1/modified$" .git/trace &&
grep "^event: dir2/modified$" .git/trace &&
grep "^event: modified$" .git/trace &&
@@ -442,10 +480,12 @@ test_expect_success 'create some files' '
start_daemon --tf "$PWD/.git/trace" &&
- create_files &&
+ wait_for_update create_files "$PWD/.git/trace" &&
test-tool fsmonitor-client query --token 0 &&
+ test_might_fail git fsmonitor--daemon stop &&
+
grep "^event: dir1/new$" .git/trace &&
grep "^event: dir2/new$" .git/trace &&
grep "^event: new$" .git/trace
@@ -456,10 +496,12 @@ test_expect_success 'delete some files' '
start_daemon --tf "$PWD/.git/trace" &&
- delete_files &&
+ wait_for_update delete_files "$PWD/.git/trace" &&
test-tool fsmonitor-client query --token 0 &&
+ test_might_fail git fsmonitor--daemon stop &&
+
grep "^event: dir1/delete$" .git/trace &&
grep "^event: dir2/delete$" .git/trace &&
grep "^event: delete$" .git/trace
@@ -470,10 +512,12 @@ test_expect_success 'rename some files' '
start_daemon --tf "$PWD/.git/trace" &&
- rename_files &&
+ wait_for_update rename_files "$PWD/.git/trace" &&
test-tool fsmonitor-client query --token 0 &&
+ test_might_fail git fsmonitor--daemon stop &&
+
grep "^event: dir1/rename$" .git/trace &&
grep "^event: dir2/rename$" .git/trace &&
grep "^event: rename$" .git/trace &&
@@ -487,23 +531,42 @@ test_expect_success 'rename directory' '
start_daemon --tf "$PWD/.git/trace" &&
- mv dirtorename dirrenamed &&
+ wait_for_update rename_directory "$PWD/.git/trace" &&
test-tool fsmonitor-client query --token 0 &&
+ test_might_fail git fsmonitor--daemon stop &&
+
grep "^event: dirtorename/*$" .git/trace &&
grep "^event: dirrenamed/*$" .git/trace
'
+test_expect_success 'rename directory file' '
+ test_when_finished clean_up_repo_and_stop_daemon &&
+
+ start_daemon --tf "$PWD/.git/trace" &&
+
+ wait_for_update rename_directory_file "$PWD/.git/trace" &&
+
+ test-tool fsmonitor-client query --token 0 &&
+
+ test_might_fail git fsmonitor--daemon stop &&
+
+ grep "^event: dirtorename/*$" .git/trace &&
+ grep "^event: dirrenamed/*$" .git/trace &&
+ grep "^event: dirrenamed/new$" .git/trace
+'
test_expect_success 'file changes to directory' '
test_when_finished clean_up_repo_and_stop_daemon &&
start_daemon --tf "$PWD/.git/trace" &&
- file_to_directory &&
+ wait_for_update file_to_directory "$PWD/.git/trace" &&
test-tool fsmonitor-client query --token 0 &&
+ test_might_fail git fsmonitor--daemon stop &&
+
grep "^event: delete$" .git/trace &&
grep "^event: delete/new$" .git/trace
'
@@ -513,10 +576,12 @@ test_expect_success 'directory changes to a file' '
start_daemon --tf "$PWD/.git/trace" &&
- directory_to_file &&
+ wait_for_update directory_to_file "$PWD/.git/trace" &&
test-tool fsmonitor-client query --token 0 &&
+ test_might_fail git fsmonitor--daemon stop &&
+
grep "^event: dir1$" .git/trace
'
@@ -561,7 +626,7 @@ test_expect_success 'flush cached data' '
test-tool -C test_flush fsmonitor-client query --token "builtin:test_00000002:0" >actual_2 &&
nul_to_q <actual_2 >actual_q2 &&
- grep "^builtin:test_00000002:0Q$" actual_q2 &&
+ grep "^builtin:test_00000002:[0-1]Q$" actual_q2 &&
>test_flush/file_3 &&
@@ -732,7 +797,8 @@ u_values="$u1 $u2"
for u in $u_values
do
test_expect_success "unicode in repo root path: $u" '
- test_when_finished "stop_daemon_delete_repo $u" &&
+ test_when_finished \
+ "stop_daemon_delete_repo `echo "$u" | sed 's:x:\\\\\\\\\\\\\\x:g'`" &&
git init "$u" &&
echo 1 >"$u"/file1 &&
@@ -818,8 +884,7 @@ test_expect_success 'submodule setup' '
'
test_expect_success 'submodule always visited' '
- test_when_finished "git -C super fsmonitor--daemon stop; \
- rm -rf super; \
+ test_when_finished "rm -rf super; \
rm -rf sub" &&
create_super super &&
@@ -887,7 +952,8 @@ have_t2_error_event () {
}
test_expect_success "stray submodule super-prefix warning" '
- test_when_finished "rm -rf super; \
+ test_when_finished "git -C super/dir_1/dir_2/sub fsmonitor--daemon stop; \
+ rm -rf super; \
rm -rf sub; \
rm super-sub.trace" &&
--
gitgitgadget
^ permalink raw reply related [flat|nested] 89+ messages in thread* [PATCH v3 6/6] fsmonitor: update doc for Linux
2022-11-16 23:23 ` [PATCH v3 0/6] " Eric DeCosta via GitGitGadget
` (4 preceding siblings ...)
2022-11-16 23:23 ` [PATCH v3 5/6] fsmonitor: test updates Eric DeCosta via GitGitGadget
@ 2022-11-16 23:23 ` Eric DeCosta via GitGitGadget
2022-11-16 23:27 ` [PATCH v3 0/6] fsmonitor: Implement fsmonitor " Taylor Blau
2022-11-23 19:00 ` [PATCH v4 " Eric DeCosta via GitGitGadget
7 siblings, 0 replies; 89+ messages in thread
From: Eric DeCosta via GitGitGadget @ 2022-11-16 23:23 UTC (permalink / raw)
To: git
Cc: Eric Sunshine, Ævar Arnfjörð Bjarmason, Glen Choo,
Johannes Schindelin, Eric DeCosta, Eric DeCosta
From: Eric DeCosta <edecosta@mathworks.com>
Update the documentation for Linux.
Signed-off-by: Eric DeCosta <edecosta@mathworks.com>
---
Documentation/config/fsmonitor--daemon.txt | 4 ++--
Documentation/git-fsmonitor--daemon.txt | 24 ++++++++++++++--------
2 files changed, 18 insertions(+), 10 deletions(-)
diff --git a/Documentation/config/fsmonitor--daemon.txt b/Documentation/config/fsmonitor--daemon.txt
index c225c6c9e74..2cafb040d96 100644
--- a/Documentation/config/fsmonitor--daemon.txt
+++ b/Documentation/config/fsmonitor--daemon.txt
@@ -4,8 +4,8 @@ fsmonitor.allowRemote::
behavior. Only respected when `core.fsmonitor` is set to `true`.
fsmonitor.socketDir::
- This Mac OS-specific option, if set, specifies the directory in
+ Mac OS and Linux-specific option. If set, specifies the directory in
which to create the Unix domain socket used for communication
between the fsmonitor daemon and various Git commands. The directory must
- reside on a native Mac OS filesystem. Only respected when `core.fsmonitor`
+ reside on a native filesystem. Only respected when `core.fsmonitor`
is set to `true`.
diff --git a/Documentation/git-fsmonitor--daemon.txt b/Documentation/git-fsmonitor--daemon.txt
index 8238eadb0e1..c2b08229c74 100644
--- a/Documentation/git-fsmonitor--daemon.txt
+++ b/Documentation/git-fsmonitor--daemon.txt
@@ -76,23 +76,31 @@ repositories; this may be overridden by setting `fsmonitor.allowRemote` to
correctly with all network-mounted repositories and such use is considered
experimental.
-On Mac OS, the inter-process communication (IPC) between various Git
+On Linux and Mac OS, the inter-process communication (IPC) between various Git
commands and the fsmonitor daemon is done via a Unix domain socket (UDS) -- a
-special type of file -- which is supported by native Mac OS filesystems,
-but not on network-mounted filesystems, NTFS, or FAT32. Other filesystems
-may or may not have the needed support; the fsmonitor daemon is not guaranteed
-to work with these filesystems and such use is considered experimental.
+special type of file -- which is supported by many native Linux and Mac OS
+filesystems, but not on network-mounted filesystems, NTFS, or FAT32. Other
+filesystems may or may not have the needed support; the fsmonitor daemon is not
+guaranteed to work with these filesystems and such use is considered
+experimental.
By default, the socket is created in the `.git` directory, however, if the
`.git` directory is on a network-mounted filesystem, it will be instead be
created at `$HOME/.git-fsmonitor-*` unless `$HOME` itself is on a
network-mounted filesystem in which case you must set the configuration
-variable `fsmonitor.socketDir` to the path of a directory on a Mac OS native
+variable `fsmonitor.socketDir` to the path of a directory on a native
filesystem in which to create the socket file.
If none of the above directories (`.git`, `$HOME`, or `fsmonitor.socketDir`)
-is on a native Mac OS file filesystem the fsmonitor daemon will report an
-error that will cause the daemon and the currently running command to exit.
+is on a native Linux or Mac OS filesystem the fsmonitor daemon will report
+an error that will cause the daemon to exit and the currently running command
+to issue a warning.
+
+On Linux, the fsmonitor daemon registers a watch for each directory in the
+repository. The default per-user limit for the number of watches on most Linux
+systems is 8192. This may not be sufficient for large repositories or if
+multiple instances of the fsmonitor daemon are running.
+See https://watchexec.github.io/docs/inotify-limits.html[Linux inotify limits] for more information.
CONFIGURATION
-------------
--
gitgitgadget
^ permalink raw reply related [flat|nested] 89+ messages in thread* Re: [PATCH v3 0/6] fsmonitor: Implement fsmonitor for Linux
2022-11-16 23:23 ` [PATCH v3 0/6] " Eric DeCosta via GitGitGadget
` (5 preceding siblings ...)
2022-11-16 23:23 ` [PATCH v3 6/6] fsmonitor: update doc for Linux Eric DeCosta via GitGitGadget
@ 2022-11-16 23:27 ` Taylor Blau
2022-11-23 19:00 ` [PATCH v4 " Eric DeCosta via GitGitGadget
7 siblings, 0 replies; 89+ messages in thread
From: Taylor Blau @ 2022-11-16 23:27 UTC (permalink / raw)
To: Eric DeCosta via GitGitGadget
Cc: git, Eric Sunshine, Ævar Arnfjörð Bjarmason,
Glen Choo, Johannes Schindelin, Eric DeCosta
[+cc Jeff Hostetler]
On Wed, Nov 16, 2022 at 11:23:33PM +0000, Eric DeCosta via GitGitGadget wrote:
> Eric DeCosta (6):
> fsmonitor: prepare to share code between Mac OS and Linux
> fsmonitor: determine if filesystem is local or remote
> fsmonitor: implement filesystem change listener for Linux
> fsmonitor: enable fsmonitor for Linux
> fsmonitor: test updates
> fsmonitor: update doc for Linux
Thanks, I picked these up. Let's add Jeff Hostetler (cc'd) to the chain
and see if he has any thoughts.
Thanks,
Taylor
^ permalink raw reply [flat|nested] 89+ messages in thread* [PATCH v4 0/6] fsmonitor: Implement fsmonitor for Linux
2022-11-16 23:23 ` [PATCH v3 0/6] " Eric DeCosta via GitGitGadget
` (6 preceding siblings ...)
2022-11-16 23:27 ` [PATCH v3 0/6] fsmonitor: Implement fsmonitor " Taylor Blau
@ 2022-11-23 19:00 ` Eric DeCosta via GitGitGadget
2022-11-23 19:00 ` [PATCH v4 1/6] fsmonitor: prepare to share code between Mac OS and Linux Eric DeCosta via GitGitGadget
` (6 more replies)
7 siblings, 7 replies; 89+ messages in thread
From: Eric DeCosta via GitGitGadget @ 2022-11-23 19:00 UTC (permalink / raw)
To: git
Cc: Eric Sunshine, Ævar Arnfjörð Bjarmason, Glen Choo,
Johannes Schindelin, Taylor Blau, Eric DeCosta
Goal is to deliver fsmonitor for Linux that is on par with fsmonitor for
Windows and Mac OS.
This patch set builds upon previous work for done for Windows and Mac OS to
implement a fsmonitor back-end for Linux based on the Linux inotify API.
inotify differs significantly from the equivalent Windows and Mac OS APIs in
that a watch must be registered for every directory of interest (rather than
a singular watch at the root of the directory tree) and special care must be
taken to handle directory renames correctly.
More information about inotify:
https://man7.org/linux/man-pages/man7/inotify.7.html
v3 differs from v2:
* Avoid potential entanglements with GPLv3
* Classify a reasonable set of filesystems as being remote
v2 differs from v1:
* Prior work for Windows and Mac OS has been merged to master, reducing the
patch set from 12 to 6 patches
* Code review feedback
* Identified and resolved race condition revealed by CI test system, see
"Limitations and caveats" regarding monitoring of directory trees from
the man page, above
* Apologies for being away from this for so long, but my attention was
needed elsewhere
v1 differs from v0:
* Code review feedback
* Update how and which code can be shared between Mac OS and Linux
* Increase polling frequency to every 1ms (matches Mac OS)
* Updates to t7527 to improve test stability
Eric DeCosta (6):
fsmonitor: prepare to share code between Mac OS and Linux
fsmonitor: determine if filesystem is local or remote
fsmonitor: implement filesystem change listener for Linux
fsmonitor: enable fsmonitor for Linux
fsmonitor: test updates
fsmonitor: update doc for Linux
Documentation/config/fsmonitor--daemon.txt | 4 +-
Documentation/git-fsmonitor--daemon.txt | 24 +-
compat/fsmonitor/fsm-health-linux.c | 24 +
compat/fsmonitor/fsm-ipc-darwin.c | 53 +-
compat/fsmonitor/fsm-ipc-linux.c | 1 +
compat/fsmonitor/fsm-ipc-unix.c | 52 ++
compat/fsmonitor/fsm-listen-linux.c | 676 +++++++++++++++++++++
compat/fsmonitor/fsm-path-utils-linux.c | 186 ++++++
compat/fsmonitor/fsm-settings-darwin.c | 63 +-
compat/fsmonitor/fsm-settings-linux.c | 1 +
compat/fsmonitor/fsm-settings-unix.c | 61 ++
config.mak.uname | 8 +
contrib/buildsystems/CMakeLists.txt | 11 +-
t/t7527-builtin-fsmonitor.sh | 94 ++-
14 files changed, 1119 insertions(+), 139 deletions(-)
create mode 100644 compat/fsmonitor/fsm-health-linux.c
create mode 100644 compat/fsmonitor/fsm-ipc-linux.c
create mode 100644 compat/fsmonitor/fsm-ipc-unix.c
create mode 100644 compat/fsmonitor/fsm-listen-linux.c
create mode 100644 compat/fsmonitor/fsm-path-utils-linux.c
create mode 100644 compat/fsmonitor/fsm-settings-linux.c
create mode 100644 compat/fsmonitor/fsm-settings-unix.c
base-commit: 319605f8f00e402f3ea758a02c63534ff800a711
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1352%2Fedecosta-mw%2Ffsmonitor_linux-v4
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1352/edecosta-mw/fsmonitor_linux-v4
Pull-Request: https://github.com/git/git/pull/1352
Range-diff vs v3:
1: 99d684c7bdf = 1: 99d684c7bdf fsmonitor: prepare to share code between Mac OS and Linux
2: aa405379891 ! 2: e53fc077540 fsmonitor: determine if filesystem is local or remote
@@ compat/fsmonitor/fsm-path-utils-linux.c (new)
+#include <errno.h>
+#include <mntent.h>
+#include <sys/mount.h>
++#include <sys/vfs.h>
+#include <sys/statvfs.h>
+
-+/*
-+ * https://github.com/coreutils/gnulib/blob/master/lib/mountlist.c
-+ */
-+#ifndef ME_REMOTE
-+/* A file system is "remote" if its Fs_name contains a ':'
-+ or if (it is of type (smbfs or cifs) and its Fs_name starts with '//')
-+ or if it is of any other of the listed types
-+ or Fs_name is equal to "-hosts" (used by autofs to mount remote fs).
-+ "VM" file systems like prl_fs or vboxsf are not considered remote here. */
-+# define ME_REMOTE(Fs_name, Fs_type) \
-+ (strchr (Fs_name, ':') != NULL \
-+ || ((Fs_name)[0] == '/' \
-+ && (Fs_name)[1] == '/' \
-+ && (strcmp (Fs_type, "smbfs") == 0 \
-+ || strcmp (Fs_type, "smb3") == 0 \
-+ || strcmp (Fs_type, "cifs") == 0)) \
-+ || strcmp (Fs_type, "acfs") == 0 \
-+ || strcmp (Fs_type, "afs") == 0 \
-+ || strcmp (Fs_type, "coda") == 0 \
-+ || strcmp (Fs_type, "auristorfs") == 0 \
-+ || strcmp (Fs_type, "fhgfs") == 0 \
-+ || strcmp (Fs_type, "gpfs") == 0 \
-+ || strcmp (Fs_type, "ibrix") == 0 \
-+ || strcmp (Fs_type, "ocfs2") == 0 \
-+ || strcmp (Fs_type, "vxfs") == 0 \
-+ || strcmp ("-hosts", Fs_name) == 0)
-+#endif
++static int is_remote_fs(const char* path) {
++ struct statfs fs;
++
++ if (statfs(path, &fs)) {
++ error_errno(_("statfs('%s') failed"), path);
++ return -1;
++ }
++
++ switch (fs.f_type) {
++ case 0x61636673: /* ACFS */
++ case 0x5346414F: /* AFS */
++ case 0x00C36400: /* CEPH */
++ case 0xFF534D42: /* CIFS */
++ case 0x73757245: /* CODA */
++ case 0x19830326: /* FHGFS */
++ case 0x1161970: /* GFS */
++ case 0x47504653: /* GPFS */
++ case 0x013111A8: /* IBRIX */
++ case 0x6B414653: /* KAFS */
++ case 0x0BD00BD0: /* LUSTRE */
++ case 0x564C: /* NCP */
++ case 0x6969: /* NFS */
++ case 0x6E667364: /* NFSD */
++ case 0x7461636f: /* OCFS2 */
++ case 0xAAD7AAEA: /* PANFS */
++ case 0x517B: /* SMB */
++ case 0xBEEFDEAD: /* SNFS */
++ case 0xFE534D42: /* SMB2 */
++ case 0xBACBACBC: /* VMHGFS */
++ case 0xA501FCF5: /* VXFS */
++ return 1;
++ default:
++ break;
++ }
++
++ return 0;
++}
+
+static int find_mount(const char *path, const struct statvfs *fs,
+ struct mntent *ent)
@@ compat/fsmonitor/fsm-path-utils-linux.c (new)
+ "statvfs('%s') [flags 0x%08lx] '%s' '%s'",
+ path, fs.f_flag, ment.mnt_type, ment.mnt_fsname);
+
-+ fs_info->is_remote = ME_REMOTE(ment.mnt_fsname, ment.mnt_type);
++ fs_info->is_remote = is_remote_fs(ment.mnt_dir);
+ fs_info->typename = ment.mnt_fsname;
+ free(ment.mnt_dir);
+ free(ment.mnt_type);
+
++ if (fs_info->is_remote < 0) {
++ free(ment.mnt_fsname);
++ return -1;
++ }
++
+ trace_printf_key(&trace_fsmonitor,
+ "'%s' is_remote: %d",
+ path, fs_info->is_remote);
++
+ return 0;
+}
+
3: c2e5a7201aa = 3: 80282efef57 fsmonitor: implement filesystem change listener for Linux
4: 05f5b2dd4fb = 4: cb03803e355 fsmonitor: enable fsmonitor for Linux
5: 754355ca44f = 5: 8d9d469b356 fsmonitor: test updates
6: f56175e097a = 6: 5afd03fa6ca fsmonitor: update doc for Linux
--
gitgitgadget
^ permalink raw reply [flat|nested] 89+ messages in thread* [PATCH v4 1/6] fsmonitor: prepare to share code between Mac OS and Linux
2022-11-23 19:00 ` [PATCH v4 " Eric DeCosta via GitGitGadget
@ 2022-11-23 19:00 ` Eric DeCosta via GitGitGadget
2022-11-23 19:00 ` [PATCH v4 2/6] fsmonitor: determine if filesystem is local or remote Eric DeCosta via GitGitGadget
` (5 subsequent siblings)
6 siblings, 0 replies; 89+ messages in thread
From: Eric DeCosta via GitGitGadget @ 2022-11-23 19:00 UTC (permalink / raw)
To: git
Cc: Eric Sunshine, Ævar Arnfjörð Bjarmason, Glen Choo,
Johannes Schindelin, Taylor Blau, Eric DeCosta, Eric DeCosta
From: Eric DeCosta <edecosta@mathworks.com>
Linux and Mac OS can share some of the code originally developed for Mac OS.
Mac OS and Linux can share fsm-ipc-unix.c and fsm-settings-unix.c
Signed-off-by: Eric DeCosta <edecosta@mathworks.com>
---
compat/fsmonitor/fsm-health-linux.c | 24 ++++++++++
compat/fsmonitor/fsm-ipc-darwin.c | 53 +---------------------
compat/fsmonitor/fsm-ipc-linux.c | 1 +
compat/fsmonitor/fsm-ipc-unix.c | 52 +++++++++++++++++++++
compat/fsmonitor/fsm-settings-darwin.c | 63 +-------------------------
compat/fsmonitor/fsm-settings-linux.c | 1 +
compat/fsmonitor/fsm-settings-unix.c | 61 +++++++++++++++++++++++++
7 files changed, 141 insertions(+), 114 deletions(-)
create mode 100644 compat/fsmonitor/fsm-health-linux.c
create mode 100644 compat/fsmonitor/fsm-ipc-linux.c
create mode 100644 compat/fsmonitor/fsm-ipc-unix.c
create mode 100644 compat/fsmonitor/fsm-settings-linux.c
create mode 100644 compat/fsmonitor/fsm-settings-unix.c
diff --git a/compat/fsmonitor/fsm-health-linux.c b/compat/fsmonitor/fsm-health-linux.c
new file mode 100644
index 00000000000..b9f709e8548
--- /dev/null
+++ b/compat/fsmonitor/fsm-health-linux.c
@@ -0,0 +1,24 @@
+#include "cache.h"
+#include "config.h"
+#include "fsmonitor.h"
+#include "fsm-health.h"
+#include "fsmonitor--daemon.h"
+
+int fsm_health__ctor(struct fsmonitor_daemon_state *state)
+{
+ return 0;
+}
+
+void fsm_health__dtor(struct fsmonitor_daemon_state *state)
+{
+ return;
+}
+
+void fsm_health__loop(struct fsmonitor_daemon_state *state)
+{
+ return;
+}
+
+void fsm_health__stop_async(struct fsmonitor_daemon_state *state)
+{
+}
diff --git a/compat/fsmonitor/fsm-ipc-darwin.c b/compat/fsmonitor/fsm-ipc-darwin.c
index d67b0ee50d3..4c3c92081ee 100644
--- a/compat/fsmonitor/fsm-ipc-darwin.c
+++ b/compat/fsmonitor/fsm-ipc-darwin.c
@@ -1,52 +1 @@
-#include "cache.h"
-#include "config.h"
-#include "strbuf.h"
-#include "fsmonitor.h"
-#include "fsmonitor-ipc.h"
-#include "fsmonitor-path-utils.h"
-
-static GIT_PATH_FUNC(fsmonitor_ipc__get_default_path, "fsmonitor--daemon.ipc")
-
-const char *fsmonitor_ipc__get_path(struct repository *r)
-{
- static const char *ipc_path = NULL;
- git_SHA_CTX sha1ctx;
- char *sock_dir = NULL;
- struct strbuf ipc_file = STRBUF_INIT;
- unsigned char hash[GIT_MAX_RAWSZ];
-
- if (!r)
- BUG("No repository passed into fsmonitor_ipc__get_path");
-
- if (ipc_path)
- return ipc_path;
-
-
- /* By default the socket file is created in the .git directory */
- if (fsmonitor__is_fs_remote(r->gitdir) < 1) {
- ipc_path = fsmonitor_ipc__get_default_path();
- return ipc_path;
- }
-
- git_SHA1_Init(&sha1ctx);
- git_SHA1_Update(&sha1ctx, r->worktree, strlen(r->worktree));
- git_SHA1_Final(hash, &sha1ctx);
-
- repo_config_get_string(r, "fsmonitor.socketdir", &sock_dir);
-
- /* Create the socket file in either socketDir or $HOME */
- if (sock_dir && *sock_dir) {
- strbuf_addf(&ipc_file, "%s/.git-fsmonitor-%s",
- sock_dir, hash_to_hex(hash));
- } else {
- strbuf_addf(&ipc_file, "~/.git-fsmonitor-%s", hash_to_hex(hash));
- }
- free(sock_dir);
-
- ipc_path = interpolate_path(ipc_file.buf, 1);
- if (!ipc_path)
- die(_("Invalid path: %s"), ipc_file.buf);
-
- strbuf_release(&ipc_file);
- return ipc_path;
-}
+#include "fsm-ipc-unix.c"
diff --git a/compat/fsmonitor/fsm-ipc-linux.c b/compat/fsmonitor/fsm-ipc-linux.c
new file mode 100644
index 00000000000..4c3c92081ee
--- /dev/null
+++ b/compat/fsmonitor/fsm-ipc-linux.c
@@ -0,0 +1 @@
+#include "fsm-ipc-unix.c"
diff --git a/compat/fsmonitor/fsm-ipc-unix.c b/compat/fsmonitor/fsm-ipc-unix.c
new file mode 100644
index 00000000000..d67b0ee50d3
--- /dev/null
+++ b/compat/fsmonitor/fsm-ipc-unix.c
@@ -0,0 +1,52 @@
+#include "cache.h"
+#include "config.h"
+#include "strbuf.h"
+#include "fsmonitor.h"
+#include "fsmonitor-ipc.h"
+#include "fsmonitor-path-utils.h"
+
+static GIT_PATH_FUNC(fsmonitor_ipc__get_default_path, "fsmonitor--daemon.ipc")
+
+const char *fsmonitor_ipc__get_path(struct repository *r)
+{
+ static const char *ipc_path = NULL;
+ git_SHA_CTX sha1ctx;
+ char *sock_dir = NULL;
+ struct strbuf ipc_file = STRBUF_INIT;
+ unsigned char hash[GIT_MAX_RAWSZ];
+
+ if (!r)
+ BUG("No repository passed into fsmonitor_ipc__get_path");
+
+ if (ipc_path)
+ return ipc_path;
+
+
+ /* By default the socket file is created in the .git directory */
+ if (fsmonitor__is_fs_remote(r->gitdir) < 1) {
+ ipc_path = fsmonitor_ipc__get_default_path();
+ return ipc_path;
+ }
+
+ git_SHA1_Init(&sha1ctx);
+ git_SHA1_Update(&sha1ctx, r->worktree, strlen(r->worktree));
+ git_SHA1_Final(hash, &sha1ctx);
+
+ repo_config_get_string(r, "fsmonitor.socketdir", &sock_dir);
+
+ /* Create the socket file in either socketDir or $HOME */
+ if (sock_dir && *sock_dir) {
+ strbuf_addf(&ipc_file, "%s/.git-fsmonitor-%s",
+ sock_dir, hash_to_hex(hash));
+ } else {
+ strbuf_addf(&ipc_file, "~/.git-fsmonitor-%s", hash_to_hex(hash));
+ }
+ free(sock_dir);
+
+ ipc_path = interpolate_path(ipc_file.buf, 1);
+ if (!ipc_path)
+ die(_("Invalid path: %s"), ipc_file.buf);
+
+ strbuf_release(&ipc_file);
+ return ipc_path;
+}
diff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c
index 6abbc7af3ab..14baf9f0603 100644
--- a/compat/fsmonitor/fsm-settings-darwin.c
+++ b/compat/fsmonitor/fsm-settings-darwin.c
@@ -1,62 +1 @@
-#include "config.h"
-#include "fsmonitor.h"
-#include "fsmonitor-ipc.h"
-#include "fsmonitor-settings.h"
-#include "fsmonitor-path-utils.h"
-
- /*
- * For the builtin FSMonitor, we create the Unix domain socket for the
- * IPC in the .git directory. If the working directory is remote,
- * then the socket will be created on the remote file system. This
- * can fail if the remote file system does not support UDS file types
- * (e.g. smbfs to a Windows server) or if the remote kernel does not
- * allow a non-local process to bind() the socket. (These problems
- * could be fixed by moving the UDS out of the .git directory and to a
- * well-known local directory on the client machine, but care should
- * be taken to ensure that $HOME is actually local and not a managed
- * file share.)
- *
- * FAT32 and NTFS working directories are problematic too.
- *
- * The builtin FSMonitor uses a Unix domain socket in the .git
- * directory for IPC. These Windows drive formats do not support
- * Unix domain sockets, so mark them as incompatible for the daemon.
- *
- */
-static enum fsmonitor_reason check_uds_volume(struct repository *r)
-{
- struct fs_info fs;
- const char *ipc_path = fsmonitor_ipc__get_path(r);
- struct strbuf path = STRBUF_INIT;
- strbuf_add(&path, ipc_path, strlen(ipc_path));
-
- if (fsmonitor__get_fs_info(dirname(path.buf), &fs) == -1) {
- strbuf_release(&path);
- return FSMONITOR_REASON_ERROR;
- }
-
- strbuf_release(&path);
-
- if (fs.is_remote ||
- !strcmp(fs.typename, "msdos") ||
- !strcmp(fs.typename, "ntfs")) {
- free(fs.typename);
- return FSMONITOR_REASON_NOSOCKETS;
- }
-
- free(fs.typename);
- return FSMONITOR_REASON_OK;
-}
-
-enum fsmonitor_reason fsm_os__incompatible(struct repository *r, int ipc)
-{
- enum fsmonitor_reason reason;
-
- if (ipc) {
- reason = check_uds_volume(r);
- if (reason != FSMONITOR_REASON_OK)
- return reason;
- }
-
- return FSMONITOR_REASON_OK;
-}
+#include "fsm-settings-unix.c"
diff --git a/compat/fsmonitor/fsm-settings-linux.c b/compat/fsmonitor/fsm-settings-linux.c
new file mode 100644
index 00000000000..14baf9f0603
--- /dev/null
+++ b/compat/fsmonitor/fsm-settings-linux.c
@@ -0,0 +1 @@
+#include "fsm-settings-unix.c"
diff --git a/compat/fsmonitor/fsm-settings-unix.c b/compat/fsmonitor/fsm-settings-unix.c
new file mode 100644
index 00000000000..a6ed32575a9
--- /dev/null
+++ b/compat/fsmonitor/fsm-settings-unix.c
@@ -0,0 +1,61 @@
+#include "config.h"
+#include "fsmonitor.h"
+#include "fsmonitor-ipc.h"
+#include "fsmonitor-path-utils.h"
+
+ /*
+ * For the builtin FSMonitor, we create the Unix domain socket for the
+ * IPC in the .git directory. If the working directory is remote,
+ * then the socket will be created on the remote file system. This
+ * can fail if the remote file system does not support UDS file types
+ * (e.g. smbfs to a Windows server) or if the remote kernel does not
+ * allow a non-local process to bind() the socket. (These problems
+ * could be fixed by moving the UDS out of the .git directory and to a
+ * well-known local directory on the client machine, but care should
+ * be taken to ensure that $HOME is actually local and not a managed
+ * file share.)
+ *
+ * FAT32 and NTFS working directories are problematic too.
+ *
+ * The builtin FSMonitor uses a Unix domain socket in the .git
+ * directory for IPC. These Windows drive formats do not support
+ * Unix domain sockets, so mark them as incompatible for the daemon.
+ *
+ */
+static enum fsmonitor_reason check_uds_volume(struct repository *r)
+{
+ struct fs_info fs;
+ const char *ipc_path = fsmonitor_ipc__get_path(r);
+ struct strbuf path = STRBUF_INIT;
+ strbuf_addstr(&path, ipc_path);
+
+ if (fsmonitor__get_fs_info(dirname(path.buf), &fs) == -1) {
+ strbuf_release(&path);
+ return FSMONITOR_REASON_ERROR;
+ }
+
+ strbuf_release(&path);
+
+ if (fs.is_remote ||
+ !strcmp(fs.typename, "msdos") ||
+ !strcmp(fs.typename, "ntfs")) {
+ free(fs.typename);
+ return FSMONITOR_REASON_NOSOCKETS;
+ }
+
+ free(fs.typename);
+ return FSMONITOR_REASON_OK;
+}
+
+enum fsmonitor_reason fsm_os__incompatible(struct repository *r, int ipc)
+{
+ enum fsmonitor_reason reason;
+
+ if (ipc) {
+ reason = check_uds_volume(r);
+ if (reason != FSMONITOR_REASON_OK)
+ return reason;
+ }
+
+ return FSMONITOR_REASON_OK;
+}
--
gitgitgadget
^ permalink raw reply related [flat|nested] 89+ messages in thread* [PATCH v4 2/6] fsmonitor: determine if filesystem is local or remote
2022-11-23 19:00 ` [PATCH v4 " Eric DeCosta via GitGitGadget
2022-11-23 19:00 ` [PATCH v4 1/6] fsmonitor: prepare to share code between Mac OS and Linux Eric DeCosta via GitGitGadget
@ 2022-11-23 19:00 ` Eric DeCosta via GitGitGadget
2022-11-25 7:31 ` Junio C Hamano
2022-12-12 10:24 ` Ævar Arnfjörð Bjarmason
2022-11-23 19:00 ` [PATCH v4 3/6] fsmonitor: implement filesystem change listener for Linux Eric DeCosta via GitGitGadget
` (4 subsequent siblings)
6 siblings, 2 replies; 89+ messages in thread
From: Eric DeCosta via GitGitGadget @ 2022-11-23 19:00 UTC (permalink / raw)
To: git
Cc: Eric Sunshine, Ævar Arnfjörð Bjarmason, Glen Choo,
Johannes Schindelin, Taylor Blau, Eric DeCosta, Eric DeCosta
From: Eric DeCosta <edecosta@mathworks.com>
Compare the given path to the mounted filesystems. Find the mount that is
the longest prefix of the path (if any) and determine if that mount is on a
local or remote filesystem.
Signed-off-by: Eric DeCosta <edecosta@mathworks.com>
---
compat/fsmonitor/fsm-path-utils-linux.c | 186 ++++++++++++++++++++++++
1 file changed, 186 insertions(+)
create mode 100644 compat/fsmonitor/fsm-path-utils-linux.c
diff --git a/compat/fsmonitor/fsm-path-utils-linux.c b/compat/fsmonitor/fsm-path-utils-linux.c
new file mode 100644
index 00000000000..d3281422ebc
--- /dev/null
+++ b/compat/fsmonitor/fsm-path-utils-linux.c
@@ -0,0 +1,186 @@
+#include "fsmonitor.h"
+#include "fsmonitor-path-utils.h"
+#include <errno.h>
+#include <mntent.h>
+#include <sys/mount.h>
+#include <sys/vfs.h>
+#include <sys/statvfs.h>
+
+static int is_remote_fs(const char* path) {
+ struct statfs fs;
+
+ if (statfs(path, &fs)) {
+ error_errno(_("statfs('%s') failed"), path);
+ return -1;
+ }
+
+ switch (fs.f_type) {
+ case 0x61636673: /* ACFS */
+ case 0x5346414F: /* AFS */
+ case 0x00C36400: /* CEPH */
+ case 0xFF534D42: /* CIFS */
+ case 0x73757245: /* CODA */
+ case 0x19830326: /* FHGFS */
+ case 0x1161970: /* GFS */
+ case 0x47504653: /* GPFS */
+ case 0x013111A8: /* IBRIX */
+ case 0x6B414653: /* KAFS */
+ case 0x0BD00BD0: /* LUSTRE */
+ case 0x564C: /* NCP */
+ case 0x6969: /* NFS */
+ case 0x6E667364: /* NFSD */
+ case 0x7461636f: /* OCFS2 */
+ case 0xAAD7AAEA: /* PANFS */
+ case 0x517B: /* SMB */
+ case 0xBEEFDEAD: /* SNFS */
+ case 0xFE534D42: /* SMB2 */
+ case 0xBACBACBC: /* VMHGFS */
+ case 0xA501FCF5: /* VXFS */
+ return 1;
+ default:
+ break;
+ }
+
+ return 0;
+}
+
+static int find_mount(const char *path, const struct statvfs *fs,
+ struct mntent *ent)
+{
+ const char *const mounts = "/proc/mounts";
+ const char *rp = real_pathdup(path, 1);
+ struct mntent *ment = NULL;
+ struct statvfs mntfs;
+ FILE *fp;
+ int found = 0;
+ int dlen, plen, flen = 0;
+
+ ent->mnt_fsname = NULL;
+ ent->mnt_dir = NULL;
+ ent->mnt_type = NULL;
+
+ fp = setmntent(mounts, "r");
+ if (!fp) {
+ error_errno(_("setmntent('%s') failed"), mounts);
+ return -1;
+ }
+
+ plen = strlen(rp);
+
+ /* read all the mount information and compare to path */
+ while ((ment = getmntent(fp)) != NULL) {
+ if (statvfs(ment->mnt_dir, &mntfs)) {
+ switch (errno) {
+ case EPERM:
+ case ESRCH:
+ case EACCES:
+ continue;
+ default:
+ error_errno(_("statvfs('%s') failed"), ment->mnt_dir);
+ endmntent(fp);
+ return -1;
+ }
+ }
+
+ /* is mount on the same filesystem and is a prefix of the path */
+ if ((fs->f_fsid == mntfs.f_fsid) &&
+ !strncmp(ment->mnt_dir, rp, strlen(ment->mnt_dir))) {
+ dlen = strlen(ment->mnt_dir);
+ if (dlen > plen)
+ continue;
+ /*
+ * root is always a potential match; otherwise look for
+ * directory prefix
+ */
+ if ((dlen == 1 && ment->mnt_dir[0] == '/') ||
+ (dlen > flen && (!rp[dlen] || rp[dlen] == '/'))) {
+ flen = dlen;
+ /*
+ * https://man7.org/linux/man-pages/man3/getmntent.3.html
+ *
+ * The pointer points to a static area of memory which is
+ * overwritten by subsequent calls to getmntent().
+ */
+ found = 1;
+ free(ent->mnt_fsname);
+ free(ent->mnt_dir);
+ free(ent->mnt_type);
+ ent->mnt_fsname = xstrdup(ment->mnt_fsname);
+ ent->mnt_dir = xstrdup(ment->mnt_dir);
+ ent->mnt_type = xstrdup(ment->mnt_type);
+ }
+ }
+ }
+ endmntent(fp);
+
+ if (!found)
+ return -1;
+
+ return 0;
+}
+
+int fsmonitor__get_fs_info(const char *path, struct fs_info *fs_info)
+{
+ struct mntent ment;
+ struct statvfs fs;
+
+ if (statvfs(path, &fs))
+ return error_errno(_("statvfs('%s') failed"), path);
+
+
+ if (find_mount(path, &fs, &ment) < 0) {
+ free(ment.mnt_fsname);
+ free(ment.mnt_dir);
+ free(ment.mnt_type);
+ return -1;
+ }
+
+ trace_printf_key(&trace_fsmonitor,
+ "statvfs('%s') [flags 0x%08lx] '%s' '%s'",
+ path, fs.f_flag, ment.mnt_type, ment.mnt_fsname);
+
+ fs_info->is_remote = is_remote_fs(ment.mnt_dir);
+ fs_info->typename = ment.mnt_fsname;
+ free(ment.mnt_dir);
+ free(ment.mnt_type);
+
+ if (fs_info->is_remote < 0) {
+ free(ment.mnt_fsname);
+ return -1;
+ }
+
+ trace_printf_key(&trace_fsmonitor,
+ "'%s' is_remote: %d",
+ path, fs_info->is_remote);
+
+ return 0;
+}
+
+int fsmonitor__is_fs_remote(const char *path)
+{
+ struct fs_info fs;
+
+ if (fsmonitor__get_fs_info(path, &fs))
+ return -1;
+
+ free(fs.typename);
+
+ return fs.is_remote;
+}
+
+/*
+ * No-op for now.
+ */
+int fsmonitor__get_alias(const char *path, struct alias_info *info)
+{
+ return 0;
+}
+
+/*
+ * No-op for now.
+ */
+char *fsmonitor__resolve_alias(const char *path,
+ const struct alias_info *info)
+{
+ return NULL;
+}
--
gitgitgadget
^ permalink raw reply related [flat|nested] 89+ messages in thread* Re: [PATCH v4 2/6] fsmonitor: determine if filesystem is local or remote
2022-11-23 19:00 ` [PATCH v4 2/6] fsmonitor: determine if filesystem is local or remote Eric DeCosta via GitGitGadget
@ 2022-11-25 7:31 ` Junio C Hamano
2022-12-12 10:24 ` Ævar Arnfjörð Bjarmason
1 sibling, 0 replies; 89+ messages in thread
From: Junio C Hamano @ 2022-11-25 7:31 UTC (permalink / raw)
To: Eric DeCosta via GitGitGadget
Cc: git, Eric Sunshine, Ævar Arnfjörð Bjarmason,
Glen Choo, Johannes Schindelin, Taylor Blau, Eric DeCosta
"Eric DeCosta via GitGitGadget" <gitgitgadget@gmail.com> writes:
> +static int is_remote_fs(const char* path) {
Style (asterisk sticks to the variable not to the type).
> + switch (fs.f_type) {
> + case 0x61636673: /* ACFS */
> ...
> + case 0xA501FCF5: /* VXFS */
> + return 1;
> + default:
> + break;
> + }
Align case/default to switch by de-denting one level, just like you
did the switch() statement in find_mount().
> +static int find_mount(const char *path, const struct statvfs *fs,
> + struct mntent *ent)
> +{
> + const char *const mounts = "/proc/mounts";
> + const char *rp = real_pathdup(path, 1);
Nobody seems to free() this once we are done with this function.
> + struct mntent *ment = NULL;
> + struct statvfs mntfs;
> + FILE *fp;
> + int found = 0;
> + int dlen, plen, flen = 0;
> +
> + ent->mnt_fsname = NULL;
> + ent->mnt_dir = NULL;
> + ent->mnt_type = NULL;
More on this later.
> + fp = setmntent(mounts, "r");
> + if (!fp) {
> + error_errno(_("setmntent('%s') failed"), mounts);
> + return -1;
> + }
> +
> + plen = strlen(rp);
> +
> + /* read all the mount information and compare to path */
> + while ((ment = getmntent(fp)) != NULL) {
> + if (statvfs(ment->mnt_dir, &mntfs)) {
> + switch (errno) {
> + case EPERM:
> + case ESRCH:
> + case EACCES:
> + continue;
> + default:
> + error_errno(_("statvfs('%s') failed"), ment->mnt_dir);
> + endmntent(fp);
> + return -1;
> + }
> + }
> +
> + /* is mount on the same filesystem and is a prefix of the path */
> + if ((fs->f_fsid == mntfs.f_fsid) &&
> + !strncmp(ment->mnt_dir, rp, strlen(ment->mnt_dir))) {
> + dlen = strlen(ment->mnt_dir);
> + if (dlen > plen)
> + continue;
> + /*
> + * root is always a potential match; otherwise look for
> + * directory prefix
> + */
> + if ((dlen == 1 && ment->mnt_dir[0] == '/') ||
> + (dlen > flen && (!rp[dlen] || rp[dlen] == '/'))) {
> + flen = dlen;
> + /*
> + * https://man7.org/linux/man-pages/man3/getmntent.3.html
> + *
> + * The pointer points to a static area of memory which is
> + * overwritten by subsequent calls to getmntent().
> + */
> + found = 1;
> + free(ent->mnt_fsname);
> + free(ent->mnt_dir);
> + free(ent->mnt_type);
> + ent->mnt_fsname = xstrdup(ment->mnt_fsname);
> + ent->mnt_dir = xstrdup(ment->mnt_dir);
> + ent->mnt_type = xstrdup(ment->mnt_type);
> + }
So more than one mount entries could match the given path. This
loop implements "the last one wins", but is that a sensible thing to
do? Shouldn't it be more like "the longest one, being the most
specific, wins"? If /usr mounts on / and /usr/local mounts on /usr,
asking for /usr/local/me would want to discover that it is on
the filesystem mounted at /usr/local regardless of the order in
which getmntent() returns the entries, no?
> + }
> + }
> + endmntent(fp);
> +
> + if (!found)
> + return -1;
> +
> + return 0;
> +}
> +
> +int fsmonitor__get_fs_info(const char *path, struct fs_info *fs_info)
> +{
> + struct mntent ment;
> + struct statvfs fs;
> +
> + if (statvfs(path, &fs))
> + return error_errno(_("statvfs('%s') failed"), path);
> +
> +
> + if (find_mount(path, &fs, &ment) < 0) {
> + free(ment.mnt_fsname);
> + free(ment.mnt_dir);
> + free(ment.mnt_type);
It is a good idea to free, but I _think_ the code is easier to
follow if you _clear_ ment before calling find_mount(), not in
find_mount().
> + return -1;
> + }
> +
> + trace_printf_key(&trace_fsmonitor,
> + "statvfs('%s') [flags 0x%08lx] '%s' '%s'",
> + path, fs.f_flag, ment.mnt_type, ment.mnt_fsname);
> +
> + fs_info->is_remote = is_remote_fs(ment.mnt_dir);
> + fs_info->typename = ment.mnt_fsname;
> + free(ment.mnt_dir);
> + free(ment.mnt_type);
> +
> + if (fs_info->is_remote < 0) {
> + free(ment.mnt_fsname);
> + return -1;
fs_info->typename just started to point into an already freed memory
at this point, which is a very safe thing to do to the caller of
this function.
Rather, perhaps delay the setting of typename after this statement,
which would be more friendly to the caller than telling them that
they are not allowed to touch the member when the function returns
negative.
> + }
> +
> + trace_printf_key(&trace_fsmonitor,
> + "'%s' is_remote: %d",
> + path, fs_info->is_remote);
> +
> + return 0;
> +}
> +
> +int fsmonitor__is_fs_remote(const char *path)
> +{
> + struct fs_info fs;
> +
> + if (fsmonitor__get_fs_info(path, &fs))
> + return -1;
> +
> + free(fs.typename);
> +
> + return fs.is_remote;
> +}
> +
> +/*
> + * No-op for now.
> + */
> +int fsmonitor__get_alias(const char *path, struct alias_info *info)
> +{
> + return 0;
> +}
> +
> +/*
> + * No-op for now.
> + */
> +char *fsmonitor__resolve_alias(const char *path,
> + const struct alias_info *info)
> +{
> + return NULL;
> +}
^ permalink raw reply [flat|nested] 89+ messages in thread* Re: [PATCH v4 2/6] fsmonitor: determine if filesystem is local or remote
2022-11-23 19:00 ` [PATCH v4 2/6] fsmonitor: determine if filesystem is local or remote Eric DeCosta via GitGitGadget
2022-11-25 7:31 ` Junio C Hamano
@ 2022-12-12 10:24 ` Ævar Arnfjörð Bjarmason
1 sibling, 0 replies; 89+ messages in thread
From: Ævar Arnfjörð Bjarmason @ 2022-12-12 10:24 UTC (permalink / raw)
To: Eric DeCosta via GitGitGadget
Cc: git, Eric Sunshine, Glen Choo, Johannes Schindelin, Taylor Blau,
Eric DeCosta
On Wed, Nov 23 2022, Eric DeCosta via GitGitGadget wrote:
> From: Eric DeCosta <edecosta@mathworks.com>
>
> Compare the given path to the mounted filesystems. Find the mount that is
> the longest prefix of the path (if any) and determine if that mount is on a
> local or remote filesystem.
>
> Signed-off-by: Eric DeCosta <edecosta@mathworks.com>
> ---
> compat/fsmonitor/fsm-path-utils-linux.c | 186 ++++++++++++++++++++++++
> 1 file changed, 186 insertions(+)
> create mode 100644 compat/fsmonitor/fsm-path-utils-linux.c
>
> diff --git a/compat/fsmonitor/fsm-path-utils-linux.c b/compat/fsmonitor/fsm-path-utils-linux.c
> new file mode 100644
> index 00000000000..d3281422ebc
> --- /dev/null
> +++ b/compat/fsmonitor/fsm-path-utils-linux.c
> @@ -0,0 +1,186 @@
> +#include "fsmonitor.h"
> +#include "fsmonitor-path-utils.h"
> +#include <errno.h>
> +#include <mntent.h>
> +#include <sys/mount.h>
> +#include <sys/vfs.h>
> +#include <sys/statvfs.h>
> +
> +static int is_remote_fs(const char* path) {
> + struct statfs fs;
> +
> + if (statfs(path, &fs)) {
> + error_errno(_("statfs('%s') failed"), path);
> + return -1;
> + }
Nit: Drop the braces and do:
if (statfs(...) == -1)
return error_errno(...)
> + switch (fs.f_type) {
> + case 0x61636673: /* ACFS */
> + case 0x5346414F: /* AFS */
> + case 0x00C36400: /* CEPH */
> + case 0xFF534D42: /* CIFS */
> + case 0x73757245: /* CODA */
> + case 0x19830326: /* FHGFS */
> + case 0x1161970: /* GFS */
> + case 0x47504653: /* GPFS */
> + case 0x013111A8: /* IBRIX */
> + case 0x6B414653: /* KAFS */
> + case 0x0BD00BD0: /* LUSTRE */
> + case 0x564C: /* NCP */
> + case 0x6969: /* NFS */
> + case 0x6E667364: /* NFSD */
> + case 0x7461636f: /* OCFS2 */
> + case 0xAAD7AAEA: /* PANFS */
> + case 0x517B: /* SMB */
> + case 0xBEEFDEAD: /* SNFS */
> + case 0xFE534D42: /* SMB2 */
> + case 0xBACBACBC: /* VMHGFS */
> + case 0xA501FCF5: /* VXFS */
So, before we'd compare against the name, but to avoid the GPLv3
copy/pasting we're now comparing against the fs.f_type.
If we are hardcoding them, our usual convention is to lower-case
hexdigits, so 0xbacbacbc not 0xBACBACBC.
But at least my statfs() manpage documents the named defines in
linux/magic.h for most of these. Why not use those?
> + return 1;
> + default:
> + break;
You could just "return 0" here, and...
> + }
> +
> + return 0;
...drop this "return 0".
> +}
> +
> +static int find_mount(const char *path, const struct statvfs *fs,
> + struct mntent *ent)
Misindentation.
> +{
> + const char *const mounts = "/proc/mounts";
> + const char *rp = real_pathdup(path, 1);
> + struct mntent *ment = NULL;
> + struct statvfs mntfs;
> + FILE *fp;
> + int found = 0;
> + int dlen, plen, flen = 0;
> +
> + ent->mnt_fsname = NULL;
> + ent->mnt_dir = NULL;
> + ent->mnt_type = NULL;
> +
> + fp = setmntent(mounts, "r");
> + if (!fp) {
> + error_errno(_("setmntent('%s') failed"), mounts);
> + return -1;
Ditto "return error_errno()"
> + }
> +
> + plen = strlen(rp);
Let's make "plen", "dlen" and "flen" a "size_t", not "int"
> +
> + /* read all the mount information and compare to path */
> + while ((ment = getmntent(fp)) != NULL) {
Drop the "!= NULL"
> + if (statvfs(ment->mnt_dir, &mntfs)) {
> + switch (errno) {
> + case EPERM:
> + case ESRCH:
> + case EACCES:
> + continue;
> + default:
> + error_errno(_("statvfs('%s') failed"), ment->mnt_dir);
> + endmntent(fp);
Shouldn't we check the endmntent() error too? Now, from the manpage the
interface is funny, and always returns 1.
But since this is linux-specific code it seems safe enough to go with it
& glibc assumptions and:
errno = 0;
endmntent(fp);
if (errno)
return error_errno(....);
I.e. it'll just call fclose(), which might set errno() on failure.
Maybe it's not worth it...
> + if (statvfs(path, &fs))
> + return error_errno(_("statvfs('%s') failed"), path);
Here you do use that "return error_errno(...)" pattern...", yay!
> +
> +
> + if (find_mount(path, &fs, &ment) < 0) {
> + free(ment.mnt_fsname);
> + free(ment.mnt_dir);
> + free(ment.mnt_type);
> + return -1;
> + }
> +
> + trace_printf_key(&trace_fsmonitor,
> + "statvfs('%s') [flags 0x%08lx] '%s' '%s'",
> + path, fs.f_flag, ment.mnt_type, ment.mnt_fsname);
> +
> + fs_info->is_remote = is_remote_fs(ment.mnt_dir);
> + fs_info->typename = ment.mnt_fsname;
> + free(ment.mnt_dir);
> + free(ment.mnt_type);
If you're going to \n\n-seperate this and the trace_printf_key() above I
think moving the second free() here to that "block" would make sense,
sinec here is the last time we use mnt_dir, but the last time we used
mnt_type was in the trace_printf_key().
But...
> +
> + if (fs_info->is_remote < 0) {
> + free(ment.mnt_fsname);
...aren't you NULL init-ing these, why not just for all of these:
goto error;
Then....
> + return -1;
> + }
> +
> + trace_printf_key(&trace_fsmonitor,
> + "'%s' is_remote: %d",
> + path, fs_info->is_remote);
> +
> + return 0;
Have this be:
int ret = -1; /* earlier */
ret = 0;
cleanup:
free(...);
free(...);
return ret;
> +}
> +
> +int fsmonitor__is_fs_remote(const char *path)
> +{
> + struct fs_info fs;
> +
> + if (fsmonitor__get_fs_info(path, &fs))
> + return -1;
> +
> + free(fs.typename);
This will segfault if you take the part through fsmonitor__get_fs_info()
where we don't have the fs.typename yet, i.e. if statfs() fails.
There's the trivial NULL-init way to work around it, but I think this
suggests a leaky abstraction. If we fail to get the fs info, then the
function itself should have free'd that, shouldn't it?
> +/*
> + * No-op for now.
> + */
> +char *fsmonitor__resolve_alias(const char *path,
> + const struct alias_info *info)
Ditto misindentatione
^ permalink raw reply [flat|nested] 89+ messages in thread
* [PATCH v4 3/6] fsmonitor: implement filesystem change listener for Linux
2022-11-23 19:00 ` [PATCH v4 " Eric DeCosta via GitGitGadget
2022-11-23 19:00 ` [PATCH v4 1/6] fsmonitor: prepare to share code between Mac OS and Linux Eric DeCosta via GitGitGadget
2022-11-23 19:00 ` [PATCH v4 2/6] fsmonitor: determine if filesystem is local or remote Eric DeCosta via GitGitGadget
@ 2022-11-23 19:00 ` Eric DeCosta via GitGitGadget
2022-12-12 10:42 ` Ævar Arnfjörð Bjarmason
2022-11-23 19:00 ` [PATCH v4 4/6] fsmonitor: enable fsmonitor " Eric DeCosta via GitGitGadget
` (3 subsequent siblings)
6 siblings, 1 reply; 89+ messages in thread
From: Eric DeCosta via GitGitGadget @ 2022-11-23 19:00 UTC (permalink / raw)
To: git
Cc: Eric Sunshine, Ævar Arnfjörð Bjarmason, Glen Choo,
Johannes Schindelin, Taylor Blau, Eric DeCosta, Eric DeCosta
From: Eric DeCosta <edecosta@mathworks.com>
Implement a filesystem change listener for Linux based on the inotify API:
https://man7.org/linux/man-pages/man7/inotify.7.html
inotify requires registering a watch on every directory in the worktree and
special handling of moves/renames.
Signed-off-by: Eric DeCosta <edecosta@mathworks.com>
---
compat/fsmonitor/fsm-listen-linux.c | 676 ++++++++++++++++++++++++++++
1 file changed, 676 insertions(+)
create mode 100644 compat/fsmonitor/fsm-listen-linux.c
diff --git a/compat/fsmonitor/fsm-listen-linux.c b/compat/fsmonitor/fsm-listen-linux.c
new file mode 100644
index 00000000000..e8548e4e009
--- /dev/null
+++ b/compat/fsmonitor/fsm-listen-linux.c
@@ -0,0 +1,676 @@
+#include "cache.h"
+#include "fsmonitor.h"
+#include "fsm-listen.h"
+#include "fsmonitor--daemon.h"
+#include <dirent.h>
+#include <fcntl.h>
+#include <sys/inotify.h>
+#include <sys/stat.h>
+
+/*
+ * Safe value to bitwise OR with rest of mask for
+ * kernels that do not support IN_MASK_CREATE
+ */
+#ifndef IN_MASK_CREATE
+#define IN_MASK_CREATE 0x00000000
+#endif
+
+enum shutdown_reason {
+ SHUTDOWN_CONTINUE = 0,
+ SHUTDOWN_STOP,
+ SHUTDOWN_ERROR,
+ SHUTDOWN_FORCE
+};
+
+struct watch_entry {
+ struct hashmap_entry ent;
+ int wd;
+ uint32_t cookie;
+ const char *dir;
+};
+
+struct rename_entry {
+ struct hashmap_entry ent;
+ time_t whence;
+ uint32_t cookie;
+ const char *dir;
+};
+
+struct fsm_listen_data {
+ int fd_inotify;
+ enum shutdown_reason shutdown;
+ struct hashmap watches;
+ struct hashmap renames;
+ struct hashmap revwatches;
+};
+
+static int watch_entry_cmp(const void *cmp_data,
+ const struct hashmap_entry *eptr,
+ const struct hashmap_entry *entry_or_key,
+ const void *keydata)
+{
+ const struct watch_entry *e1, *e2;
+
+ e1 = container_of(eptr, const struct watch_entry, ent);
+ e2 = container_of(eptr, const struct watch_entry, ent);
+ return e1->wd != e2->wd;
+}
+
+static int revwatches_entry_cmp(const void *cmp_data,
+ const struct hashmap_entry *eptr,
+ const struct hashmap_entry *entry_or_key,
+ const void *keydata)
+{
+ const struct watch_entry *e1, *e2;
+
+ e1 = container_of(eptr, const struct watch_entry, ent);
+ e2 = container_of(eptr, const struct watch_entry, ent);
+ return strcmp(e1->dir, e2->dir);
+}
+
+static int rename_entry_cmp(const void *cmp_data,
+ const struct hashmap_entry *eptr,
+ const struct hashmap_entry *entry_or_key,
+ const void *keydata)
+{
+ const struct rename_entry *e1, *e2;
+
+ e1 = container_of(eptr, const struct rename_entry, ent);
+ e2 = container_of(eptr, const struct rename_entry, ent);
+ return e1->cookie != e2->cookie;
+}
+
+/*
+ * Register an inotify watch, add watch descriptor to path mapping
+ * and the reverse mapping.
+ */
+static int add_watch(const char *path, struct fsm_listen_data *data)
+{
+ const char *interned = strintern(path);
+ struct watch_entry *w1, *w2;
+
+ /* add the inotify watch, don't allow watches to be modified */
+ int wd = inotify_add_watch(data->fd_inotify, interned,
+ (IN_ALL_EVENTS | IN_ONLYDIR | IN_MASK_CREATE)
+ ^ IN_ACCESS ^ IN_CLOSE ^ IN_OPEN);
+ if (wd < 0)
+ return error_errno("inotify_add_watch('%s') failed", interned);
+
+ /* add watch descriptor -> directory mapping */
+ CALLOC_ARRAY(w1, 1);
+ w1->wd = wd;
+ w1->dir = interned;
+ hashmap_entry_init(&w1->ent, memhash(&w1->wd, sizeof(int)));
+ hashmap_add(&data->watches, &w1->ent);
+
+ /* add directory -> watch descriptor mapping */
+ CALLOC_ARRAY(w2, 1);
+ w2->wd = wd;
+ w2->dir = interned;
+ hashmap_entry_init(&w2->ent, memhash(w2->dir, strlen(w2->dir)));
+ hashmap_add(&data->revwatches, &w2->ent);
+
+ return 0;
+}
+
+/*
+ * Remove the inotify watch, the watch descriptor to path mapping
+ * and the reverse mapping.
+ */
+static void remove_watch(struct watch_entry *w,
+ struct fsm_listen_data *data)
+{
+ struct watch_entry k1, k2, *w1, *w2;
+
+ /* remove watch, ignore error if kernel already did it */
+ if (inotify_rm_watch(data->fd_inotify, w->wd) && errno != EINVAL)
+ error_errno("inotify_rm_watch() failed");
+
+ hashmap_entry_init(&k1.ent, memhash(&w->wd, sizeof(int)));
+ w1 = hashmap_remove_entry(&data->watches, &k1, ent, NULL);
+ if (!w1)
+ BUG("Double remove of watch for '%s'", w->dir);
+
+ if (w1->cookie)
+ BUG("Removing watch for '%s' which has a pending rename", w1->dir);
+
+ hashmap_entry_init(&k2.ent, memhash(w->dir, strlen(w->dir)));
+ w2 = hashmap_remove_entry(&data->revwatches, &k2, ent, NULL);
+ if (!w2)
+ BUG("Double remove of reverse watch for '%s'", w->dir);
+
+ /* w1->dir and w2->dir are interned strings, we don't own them */
+ free(w1);
+ free(w2);
+}
+
+/*
+ * Check for stale directory renames.
+ *
+ * https://man7.org/linux/man-pages/man7/inotify.7.html
+ *
+ * Allow for some small timeout to account for the fact that insertion of the
+ * IN_MOVED_FROM+IN_MOVED_TO event pair is not atomic, and the possibility that
+ * there may not be any IN_MOVED_TO event.
+ *
+ * If the IN_MOVED_TO event is not received within the timeout then events have
+ * been missed and the monitor is in an inconsistent state with respect to the
+ * filesystem.
+ */
+static int check_stale_dir_renames(struct hashmap *renames, time_t max_age)
+{
+ struct rename_entry *re;
+ struct hashmap_iter iter;
+
+ hashmap_for_each_entry(renames, &iter, re, ent) {
+ if (re->whence <= max_age)
+ return -1;
+ }
+ return 0;
+}
+
+/*
+ * Track pending renames.
+ *
+ * Tracking is done via a event cookie to watch descriptor mapping.
+ *
+ * A rename is not complete until matching a IN_MOVED_TO event is received
+ * for a corresponding IN_MOVED_FROM event.
+ */
+static void add_dir_rename(uint32_t cookie, const char *path,
+ struct fsm_listen_data *data)
+{
+ struct watch_entry k, *w;
+ struct rename_entry *re;
+
+ /* lookup the watch descriptor for the given path */
+ hashmap_entry_init(&k.ent, memhash(path, strlen(path)));
+ w = hashmap_get_entry(&data->revwatches, &k, ent, NULL);
+ if (!w) /* should never happen */
+ BUG("No watch for '%s'", path);
+ w->cookie = cookie;
+
+ /* add the pending rename to match against later */
+ CALLOC_ARRAY(re, 1);
+ re->dir = w->dir;
+ re->cookie = w->cookie;
+ re->whence = time(NULL);
+ hashmap_entry_init(&re->ent, memhash(&re->cookie, sizeof(uint32_t)));
+ hashmap_add(&data->renames, &re->ent);
+}
+
+/*
+ * Handle directory renames
+ *
+ * Once a IN_MOVED_TO event is received, lookup the rename tracking information
+ * via the event cookie and use this information to update the watch.
+ */
+static void rename_dir(uint32_t cookie, const char *path,
+ struct fsm_listen_data *data)
+{
+ struct rename_entry rek, *re;
+ struct watch_entry k, *w;
+
+ /* lookup a pending rename to match */
+ rek.cookie = cookie;
+ hashmap_entry_init(&rek.ent, memhash(&rek.cookie, sizeof(uint32_t)));
+ re = hashmap_get_entry(&data->renames, &rek, ent, NULL);
+ if (re) {
+ k.dir = re->dir;
+ hashmap_entry_init(&k.ent, memhash(k.dir, strlen(k.dir)));
+ w = hashmap_get_entry(&data->revwatches, &k, ent, NULL);
+ if (w) {
+ w->cookie = 0; /* rename handled */
+ remove_watch(w, data);
+ add_watch(path, data);
+ } else {
+ BUG("No matching watch");
+ }
+ } else {
+ BUG("No matching cookie");
+ }
+}
+
+/*
+ * Recursively add watches to every directory under path
+ */
+static int register_inotify(const char *path,
+ struct fsmonitor_daemon_state *state,
+ struct fsmonitor_batch *batch)
+{
+ DIR *dir;
+ const char *rel;
+ struct strbuf current = STRBUF_INIT;
+ struct dirent *de;
+ struct stat fs;
+ int ret = -1;
+
+ dir = opendir(path);
+ if (!dir)
+ return error_errno("opendir('%s') failed", path);
+
+ while ((de = readdir_skip_dot_and_dotdot(dir)) != NULL) {
+ strbuf_reset(¤t);
+ strbuf_addf(¤t, "%s/%s", path, de->d_name);
+ if (lstat(current.buf, &fs)) {
+ error_errno("lstat('%s') failed", current.buf);
+ goto failed;
+ }
+
+ /* recurse into directory */
+ if (S_ISDIR(fs.st_mode)) {
+ if (add_watch(current.buf, state->listen_data))
+ goto failed;
+ if (register_inotify(current.buf, state, batch))
+ goto failed;
+ } else if (batch) {
+ rel = current.buf + state->path_worktree_watch.len + 1;
+ trace_printf_key(&trace_fsmonitor, "explicitly adding '%s'", rel);
+ fsmonitor_batch__add_path(batch, rel);
+ }
+ }
+ ret = 0;
+
+failed:
+ strbuf_release(¤t);
+ if (closedir(dir) < 0)
+ return error_errno("closedir('%s') failed", path);
+ return ret;
+}
+
+static int em_rename_dir_from(u_int32_t mask)
+{
+ return ((mask & IN_ISDIR) && (mask & IN_MOVED_FROM));
+}
+
+static int em_rename_dir_to(u_int32_t mask)
+{
+ return ((mask & IN_ISDIR) && (mask & IN_MOVED_TO));
+}
+
+static int em_remove_watch(u_int32_t mask)
+{
+ return (mask & IN_DELETE_SELF);
+}
+
+static int em_dir_renamed(u_int32_t mask)
+{
+ return ((mask & IN_ISDIR) && (mask & IN_MOVE));
+}
+
+static int em_dir_created(u_int32_t mask)
+{
+ return ((mask & IN_ISDIR) && (mask & IN_CREATE));
+}
+
+static int em_dir_deleted(uint32_t mask)
+{
+ return ((mask & IN_ISDIR) && (mask & IN_DELETE));
+}
+
+static int em_force_shutdown(u_int32_t mask)
+{
+ return (mask & IN_UNMOUNT) || (mask & IN_Q_OVERFLOW);
+}
+
+static int em_ignore(u_int32_t mask)
+{
+ return (mask & IN_IGNORED) || (mask & IN_MOVE_SELF);
+}
+
+static void log_mask_set(const char *path, u_int32_t mask)
+{
+ struct strbuf msg = STRBUF_INIT;
+
+ if (mask & IN_ACCESS)
+ strbuf_addstr(&msg, "IN_ACCESS|");
+ if (mask & IN_MODIFY)
+ strbuf_addstr(&msg, "IN_MODIFY|");
+ if (mask & IN_ATTRIB)
+ strbuf_addstr(&msg, "IN_ATTRIB|");
+ if (mask & IN_CLOSE_WRITE)
+ strbuf_addstr(&msg, "IN_CLOSE_WRITE|");
+ if (mask & IN_CLOSE_NOWRITE)
+ strbuf_addstr(&msg, "IN_CLOSE_NOWRITE|");
+ if (mask & IN_OPEN)
+ strbuf_addstr(&msg, "IN_OPEN|");
+ if (mask & IN_MOVED_FROM)
+ strbuf_addstr(&msg, "IN_MOVED_FROM|");
+ if (mask & IN_MOVED_TO)
+ strbuf_addstr(&msg, "IN_MOVED_TO|");
+ if (mask & IN_CREATE)
+ strbuf_addstr(&msg, "IN_CREATE|");
+ if (mask & IN_DELETE)
+ strbuf_addstr(&msg, "IN_DELETE|");
+ if (mask & IN_DELETE_SELF)
+ strbuf_addstr(&msg, "IN_DELETE_SELF|");
+ if (mask & IN_MOVE_SELF)
+ strbuf_addstr(&msg, "IN_MOVE_SELF|");
+ if (mask & IN_UNMOUNT)
+ strbuf_addstr(&msg, "IN_UNMOUNT|");
+ if (mask & IN_Q_OVERFLOW)
+ strbuf_addstr(&msg, "IN_Q_OVERFLOW|");
+ if (mask & IN_IGNORED)
+ strbuf_addstr(&msg, "IN_IGNORED|");
+ if (mask & IN_ISDIR)
+ strbuf_addstr(&msg, "IN_ISDIR|");
+
+ trace_printf_key(&trace_fsmonitor, "inotify_event: '%s', mask=%#8.8x %s",
+ path, mask, msg.buf);
+
+ strbuf_release(&msg);
+}
+
+int fsm_listen__ctor(struct fsmonitor_daemon_state *state)
+{
+ int fd;
+ int ret = 0;
+ struct fsm_listen_data *data;
+
+ CALLOC_ARRAY(data, 1);
+ state->listen_data = data;
+ state->listen_error_code = -1;
+ data->shutdown = SHUTDOWN_ERROR;
+
+ fd = inotify_init1(O_NONBLOCK);
+ if (fd < 0)
+ return error_errno("inotify_init1() failed");
+
+ data->fd_inotify = fd;
+
+ hashmap_init(&data->watches, watch_entry_cmp, NULL, 0);
+ hashmap_init(&data->renames, rename_entry_cmp, NULL, 0);
+ hashmap_init(&data->revwatches, revwatches_entry_cmp, NULL, 0);
+
+ if (add_watch(state->path_worktree_watch.buf, data))
+ ret = -1;
+ else if (register_inotify(state->path_worktree_watch.buf, state, NULL))
+ ret = -1;
+ else if (state->nr_paths_watching > 1) {
+ if (add_watch(state->path_gitdir_watch.buf, data))
+ ret = -1;
+ else if (register_inotify(state->path_gitdir_watch.buf, state, NULL))
+ ret = -1;
+ }
+
+ if (!ret) {
+ state->listen_error_code = 0;
+ data->shutdown = SHUTDOWN_CONTINUE;
+ }
+
+ return ret;
+}
+
+void fsm_listen__dtor(struct fsmonitor_daemon_state *state)
+{
+ struct fsm_listen_data *data;
+ struct hashmap_iter iter;
+ struct watch_entry *w;
+ int fd;
+
+ if (!state || !state->listen_data)
+ return;
+
+ data = state->listen_data;
+ fd = data->fd_inotify;
+
+ hashmap_for_each_entry(&data->watches, &iter, w, ent) {
+ w->cookie = 0; /* ignore any pending renames */
+ remove_watch(w, data);
+ }
+ hashmap_clear(&data->watches);
+
+ hashmap_clear(&data->revwatches); /* remove_watch freed the entries */
+
+ hashmap_clear_and_free(&data->renames, struct rename_entry, ent);
+
+ FREE_AND_NULL(state->listen_data);
+
+ if (fd && (close(fd) < 0))
+ error_errno(_("closing inotify file descriptor failed"));
+}
+
+void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)
+{
+ if (!state->listen_data->shutdown)
+ state->listen_data->shutdown = SHUTDOWN_STOP;
+}
+
+/*
+ * Process a single inotify event and queue for publication.
+ */
+static int process_event(const char *path,
+ const struct inotify_event *event,
+ struct fsmonitor_batch *batch,
+ struct string_list *cookie_list,
+ struct fsmonitor_daemon_state *state)
+{
+ const char *rel;
+ const char *last_sep;
+
+ switch (fsmonitor_classify_path_absolute(state, path)) {
+ case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:
+ case IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:
+ /* Use just the filename of the cookie file. */
+ last_sep = find_last_dir_sep(path);
+ string_list_append(cookie_list,
+ last_sep ? last_sep + 1 : path);
+ break;
+ case IS_INSIDE_DOT_GIT:
+ case IS_INSIDE_GITDIR:
+ break;
+ case IS_DOT_GIT:
+ case IS_GITDIR:
+ /*
+ * If .git directory is deleted or renamed away,
+ * we have to quit.
+ */
+ if (em_dir_deleted(event->mask)) {
+ trace_printf_key(&trace_fsmonitor,
+ "event: gitdir removed");
+ state->listen_data->shutdown = SHUTDOWN_FORCE;
+ goto done;
+ }
+
+ if (em_dir_renamed(event->mask)) {
+ trace_printf_key(&trace_fsmonitor,
+ "event: gitdir renamed");
+ state->listen_data->shutdown = SHUTDOWN_FORCE;
+ goto done;
+ }
+ break;
+ case IS_WORKDIR_PATH:
+ /* normal events in the working directory */
+ if (trace_pass_fl(&trace_fsmonitor))
+ log_mask_set(path, event->mask);
+
+ rel = path + state->path_worktree_watch.len + 1;
+ fsmonitor_batch__add_path(batch, rel);
+
+ if (em_dir_deleted(event->mask))
+ break;
+
+ /* received IN_MOVE_FROM, add tracking for expected IN_MOVE_TO */
+ if (em_rename_dir_from(event->mask))
+ add_dir_rename(event->cookie, path, state->listen_data);
+
+ /* received IN_MOVE_TO, update watch to reflect new path */
+ if (em_rename_dir_to(event->mask)) {
+ rename_dir(event->cookie, path, state->listen_data);
+ if (register_inotify(path, state, batch)) {
+ state->listen_data->shutdown = SHUTDOWN_ERROR;
+ goto done;
+ }
+ }
+
+ if (em_dir_created(event->mask)) {
+ if (add_watch(path, state->listen_data)) {
+ state->listen_data->shutdown = SHUTDOWN_ERROR;
+ goto done;
+ }
+ if (register_inotify(path, state, batch)) {
+ state->listen_data->shutdown = SHUTDOWN_ERROR;
+ goto done;
+ }
+ }
+ break;
+ case IS_OUTSIDE_CONE:
+ default:
+ trace_printf_key(&trace_fsmonitor,
+ "ignoring '%s'", path);
+ break;
+ }
+ return 0;
+done:
+ return -1;
+}
+
+/*
+ * Read the inotify event stream and pre-process events before further
+ * processing and eventual publishing.
+ */
+static void handle_events(struct fsmonitor_daemon_state *state)
+{
+ /* See https://man7.org/linux/man-pages/man7/inotify.7.html */
+ char buf[4096]
+ __attribute__ ((aligned(__alignof__(struct inotify_event))));
+
+ struct hashmap watches = state->listen_data->watches;
+ struct fsmonitor_batch *batch = NULL;
+ struct string_list cookie_list = STRING_LIST_INIT_DUP;
+ struct watch_entry k, *w;
+ struct strbuf path;
+ const struct inotify_event *event;
+ int fd = state->listen_data->fd_inotify;
+ ssize_t len;
+ char *ptr, *p;
+
+ strbuf_init(&path, PATH_MAX);
+
+ for(;;) {
+ len = read(fd, buf, sizeof(buf));
+ if (len == -1 && errno != EAGAIN) {
+ error_errno(_("reading inotify message stream failed"));
+ state->listen_data->shutdown = SHUTDOWN_ERROR;
+ goto done;
+ }
+
+ /* nothing to read */
+ if (len <= 0)
+ goto done;
+
+ /* Loop over all events in the buffer. */
+ for (ptr = buf; ptr < buf + len;
+ ptr += sizeof(struct inotify_event) + event->len) {
+
+ event = (const struct inotify_event *) ptr;
+
+ if (em_ignore(event->mask))
+ continue;
+
+ /* File system was unmounted or event queue overflowed */
+ if (em_force_shutdown(event->mask)) {
+ if (trace_pass_fl(&trace_fsmonitor))
+ log_mask_set("Forcing shutdown", event->mask);
+ state->listen_data->shutdown = SHUTDOWN_FORCE;
+ goto done;
+ }
+
+ hashmap_entry_init(&k.ent, memhash(&event->wd, sizeof(int)));
+ k.wd = event->wd;
+
+ w = hashmap_get_entry(&watches, &k, ent, NULL);
+ if (!w) /* should never happen */
+ BUG("No watch for '%s'", event->name);
+
+ /* directory watch was removed */
+ if (em_remove_watch(event->mask)) {
+ remove_watch(w, state->listen_data);
+ continue;
+ }
+
+ strbuf_reset(&path);
+ strbuf_add(&path, w->dir, strlen(w->dir));
+ strbuf_addch(&path, '/');
+ strbuf_addstr(&path, event->name);
+
+ p = fsmonitor__resolve_alias(path.buf, &state->alias);
+ if (!p)
+ p = strbuf_detach(&path, NULL);
+
+ if (!batch)
+ batch = fsmonitor_batch__new();
+
+ if (process_event(p, event, batch, &cookie_list, state)) {
+ free(p);
+ goto done;
+ }
+ free(p);
+ }
+ strbuf_reset(&path);
+ fsmonitor_publish(state, batch, &cookie_list);
+ string_list_clear(&cookie_list, 0);
+ batch = NULL;
+ }
+done:
+ strbuf_release(&path);
+ fsmonitor_batch__free_list(batch);
+ string_list_clear(&cookie_list, 0);
+}
+
+/*
+ * Non-blocking read of the inotify events stream. The inotify fd is polled
+ * frequently to help minimize the number of queue overflows.
+ */
+void fsm_listen__loop(struct fsmonitor_daemon_state *state)
+{
+ int poll_num;
+ const int interval = 1000;
+ time_t checked = time(NULL);
+ struct pollfd fds[1];
+ fds[0].fd = state->listen_data->fd_inotify;
+ fds[0].events = POLLIN;
+
+ for(;;) {
+ switch (state->listen_data->shutdown) {
+ case SHUTDOWN_CONTINUE:
+ poll_num = poll(fds, 1, 1);
+ if (poll_num == -1) {
+ if (errno == EINTR)
+ continue;
+ error_errno(_("polling inotify message stream failed"));
+ state->listen_data->shutdown = SHUTDOWN_ERROR;
+ continue;
+ }
+
+ if ((time(NULL) - checked) >= interval) {
+ checked = time(NULL);
+ if (check_stale_dir_renames(&state->listen_data->renames,
+ checked - interval)) {
+ trace_printf_key(&trace_fsmonitor,
+ "Missed IN_MOVED_TO events, forcing shutdown");
+ state->listen_data->shutdown = SHUTDOWN_FORCE;
+ continue;
+ }
+ }
+
+ if (poll_num > 0 && (fds[0].revents & POLLIN))
+ handle_events(state);
+
+ continue;
+ case SHUTDOWN_ERROR:
+ state->listen_error_code = -1;
+ ipc_server_stop_async(state->ipc_server_data);
+ break;
+ case SHUTDOWN_FORCE:
+ state->listen_error_code = 0;
+ ipc_server_stop_async(state->ipc_server_data);
+ break;
+ case SHUTDOWN_STOP:
+ default:
+ state->listen_error_code = 0;
+ break;
+ }
+ return;
+ }
+}
--
gitgitgadget
^ permalink raw reply related [flat|nested] 89+ messages in thread* Re: [PATCH v4 3/6] fsmonitor: implement filesystem change listener for Linux
2022-11-23 19:00 ` [PATCH v4 3/6] fsmonitor: implement filesystem change listener for Linux Eric DeCosta via GitGitGadget
@ 2022-12-12 10:42 ` Ævar Arnfjörð Bjarmason
0 siblings, 0 replies; 89+ messages in thread
From: Ævar Arnfjörð Bjarmason @ 2022-12-12 10:42 UTC (permalink / raw)
To: Eric DeCosta via GitGitGadget
Cc: git, Eric Sunshine, Glen Choo, Johannes Schindelin, Taylor Blau,
Eric DeCosta
On Wed, Nov 23 2022, Eric DeCosta via GitGitGadget wrote:
> From: Eric DeCosta <edecosta@mathworks.com>
> [...]
> +}
> +
> +/*
> + * Remove the inotify watch, the watch descriptor to path mapping
> + * and the reverse mapping.
> + */
> +static void remove_watch(struct watch_entry *w,
> + struct fsm_listen_data *data)
> +{
> + struct watch_entry k1, k2, *w1, *w2;
> +
> + /* remove watch, ignore error if kernel already did it */
> + if (inotify_rm_watch(data->fd_inotify, w->wd) && errno != EINVAL)
> + error_errno("inotify_rm_watch() failed");
If we error_errno(), shouldn't this function have a return value?
> +
> + hashmap_entry_init(&k1.ent, memhash(&w->wd, sizeof(int)));
> + w1 = hashmap_remove_entry(&data->watches, &k1, ent, NULL);
> + if (!w1)
> + BUG("Double remove of watch for '%s'", w->dir);
> +
> + if (w1->cookie)
> + BUG("Removing watch for '%s' which has a pending rename", w1->dir);
> +
> + hashmap_entry_init(&k2.ent, memhash(w->dir, strlen(w->dir)));
> + w2 = hashmap_remove_entry(&data->revwatches, &k2, ent, NULL);
> + if (!w2)
> + BUG("Double remove of reverse watch for '%s'", w->dir);
For the BUG() additions: Start with lower-case in messages, see
CodingGuidelines. I.e. "double remove of ..." etc. Ditto below..
> [...]
> +/*
> + * Handle directory renames
> + *
> + * Once a IN_MOVED_TO event is received, lookup the rename tracking information
> + * via the event cookie and use this information to update the watch.
> + */
> +static void rename_dir(uint32_t cookie, const char *path,
> + struct fsm_listen_data *data)
> +{
> + struct rename_entry rek, *re;
> + struct watch_entry k, *w;
> +
> + /* lookup a pending rename to match */
> + rek.cookie = cookie;
> + hashmap_entry_init(&rek.ent, memhash(&rek.cookie, sizeof(uint32_t)));
> + re = hashmap_get_entry(&data->renames, &rek, ent, NULL);
> + if (re) {
> + k.dir = re->dir;
> + hashmap_entry_init(&k.ent, memhash(k.dir, strlen(k.dir)));
> + w = hashmap_get_entry(&data->revwatches, &k, ent, NULL);
> + if (w) {
> + w->cookie = 0; /* rename handled */
> + remove_watch(w, data);
> + add_watch(path, data);
Elsewhere you check the add_watch() return value, but not here?
> + } else {
> + BUG("No matching watch");
> + }
> + } else {
> + BUG("No matching cookie");
> + }
> +}
> +
> +/*
> + * Recursively add watches to every directory under path
> + */
> +static int register_inotify(const char *path,
> + struct fsmonitor_daemon_state *state,
> + struct fsmonitor_batch *batch)
> +{
> + DIR *dir;
> + const char *rel;
> + struct strbuf current = STRBUF_INIT;
> + struct dirent *de;
> + struct stat fs;
> + int ret = -1;
> +
> + dir = opendir(path);
> + if (!dir)
> + return error_errno("opendir('%s') failed", path);
> +
> + while ((de = readdir_skip_dot_and_dotdot(dir)) != NULL) {
ditto drop the "!= NULL"
> + struct strbuf msg = STRBUF_INIT;
> +
> + if (mask & IN_ACCESS)
> + strbuf_addstr(&msg, "IN_ACCESS|");
> + if (mask & IN_MODIFY)
> + strbuf_addstr(&msg, "IN_MODIFY|");
> + if (mask & IN_ATTRIB)
> + strbuf_addstr(&msg, "IN_ATTRIB|");
> + if (mask & IN_CLOSE_WRITE)
> + strbuf_addstr(&msg, "IN_CLOSE_WRITE|");
> + if (mask & IN_CLOSE_NOWRITE)
> + strbuf_addstr(&msg, "IN_CLOSE_NOWRITE|");
> + if (mask & IN_OPEN)
> + strbuf_addstr(&msg, "IN_OPEN|");
> + if (mask & IN_MOVED_FROM)
> + strbuf_addstr(&msg, "IN_MOVED_FROM|");
> + if (mask & IN_MOVED_TO)
> + strbuf_addstr(&msg, "IN_MOVED_TO|");
> + if (mask & IN_CREATE)
> + strbuf_addstr(&msg, "IN_CREATE|");
> + if (mask & IN_DELETE)
> + strbuf_addstr(&msg, "IN_DELETE|");
> + if (mask & IN_DELETE_SELF)
> + strbuf_addstr(&msg, "IN_DELETE_SELF|");
> + if (mask & IN_MOVE_SELF)
> + strbuf_addstr(&msg, "IN_MOVE_SELF|");
> + if (mask & IN_UNMOUNT)
> + strbuf_addstr(&msg, "IN_UNMOUNT|");
> + if (mask & IN_Q_OVERFLOW)
> + strbuf_addstr(&msg, "IN_Q_OVERFLOW|");
> + if (mask & IN_IGNORED)
> + strbuf_addstr(&msg, "IN_IGNORED|");
> + if (mask & IN_ISDIR)
> + strbuf_addstr(&msg, "IN_ISDIR|");
Shouldn't the very last addition to mask omit the "|"?
I think it would be worth just making this a NULL-delimited "int, const char *" array like:
{
{ IN_ACCESS, "IN_ACCESS" },
...
NULL,
};
Then looping over it, or maybe not...
> + trace_printf_key(&trace_fsmonitor, "inotify_event: '%s', mask=%#8.8x %s",
> + path, mask, msg.buf);
> +
> + strbuf_release(&msg);
> +}
> +
> +int fsm_listen__ctor(struct fsmonitor_daemon_state *state)
> +{
> + int fd;
> + int ret = 0;
> + struct fsm_listen_data *data;
> +
> + CALLOC_ARRAY(data, 1);
> + state->listen_data = data;
> + state->listen_error_code = -1;
> + data->shutdown = SHUTDOWN_ERROR;
> +
> + fd = inotify_init1(O_NONBLOCK);
> + if (fd < 0)
> + return error_errno("inotify_init1() failed");
Here you leak the "data" you just allocated on error.
> +
> + data->fd_inotify = fd;
> +
> + hashmap_init(&data->watches, watch_entry_cmp, NULL, 0);
> + hashmap_init(&data->renames, rename_entry_cmp, NULL, 0);
> + hashmap_init(&data->revwatches, revwatches_entry_cmp, NULL, 0);
> +
> + if (add_watch(state->path_worktree_watch.buf, data))
I.e. we should only avoid free()-ing it if we can successfully hand it
over to add_watch(), shouldn't we?n
> + ret = -1;
> + else if (register_inotify(state->path_worktree_watch.buf, state, NULL))
> + ret = -1;
We add {}'s to all if/else if branches if one needs it, see CodingGuidelines.
> + else if (state->nr_paths_watching > 1) {
> + if (add_watch(state->path_gitdir_watch.buf, data))
> + ret = -1;
> + else if (register_inotify(state->path_gitdir_watch.buf, state, NULL))
> + ret = -1;
Can't this be:
else if (state->nr_paths_watching > 1 &&
(add_watch(...) || register_inotify(...)))
ret = -1;
> + switch (fsmonitor_classify_path_absolute(state, path)) {
> + case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:
Don't indent the "case" label here more than the "switch", see
CodingGuidelines.
> +static void handle_events(struct fsmonitor_daemon_state *state)
> +{
> + /* See https://man7.org/linux/man-pages/man7/inotify.7.html */
Extra " " here after the "\t".
> + char buf[4096]
> + __attribute__ ((aligned(__alignof__(struct inotify_event))));
> +
> + struct hashmap watches = state->listen_data->watches;
> + struct fsmonitor_batch *batch = NULL;
> + struct string_list cookie_list = STRING_LIST_INIT_DUP;
> + struct watch_entry k, *w;
> + struct strbuf path;
> + const struct inotify_event *event;
> + int fd = state->listen_data->fd_inotify;
> + ssize_t len;
> + char *ptr, *p;
> +
> + strbuf_init(&path, PATH_MAX);
Just use "struct strbuf path = STRBUF_INIT" instead? I.e...
> +
> + for(;;) {
> + len = read(fd, buf, sizeof(buf));
> + if (len == -1 && errno != EAGAIN) {
> + error_errno(_("reading inotify message stream failed"));
> + state->listen_data->shutdown = SHUTDOWN_ERROR;
> + goto done;
> + }
> +
> + /* nothing to read */
> + if (len <= 0)
> + goto done;
> +
> + /* Loop over all events in the buffer. */
> + for (ptr = buf; ptr < buf + len;
> + ptr += sizeof(struct inotify_event) + event->len) {
> +
> + event = (const struct inotify_event *) ptr;
> +
> + if (em_ignore(event->mask))
> + continue;
> +
> + /* File system was unmounted or event queue overflowed */
> + if (em_force_shutdown(event->mask)) {
> + if (trace_pass_fl(&trace_fsmonitor))
> + log_mask_set("Forcing shutdown", event->mask);
> + state->listen_data->shutdown = SHUTDOWN_FORCE;
> + goto done;
> + }
> +
> + hashmap_entry_init(&k.ent, memhash(&event->wd, sizeof(int)));
> + k.wd = event->wd;
> +
> + w = hashmap_get_entry(&watches, &k, ent, NULL);
> + if (!w) /* should never happen */
> + BUG("No watch for '%s'", event->name);
> +
> + /* directory watch was removed */
> + if (em_remove_watch(event->mask)) {
> + remove_watch(w, state->listen_data);
> + continue;
> + }
> +
> + strbuf_reset(&path);
> + strbuf_add(&path, w->dir, strlen(w->dir));
> + strbuf_addch(&path, '/');
> + strbuf_addstr(&path, event->name);
... we may not even get to this, so we may pointlessly pre-grow it, if
we're considering the micro-optimization.
But if we do need it it'll quickly get up to size in the loop, which is
probably smaller than PATH_MAX, and paths can exceed PATH_MAX....
> + for(;;) {
> + switch (state->listen_data->shutdown) {
> + case SHUTDOWN_CONTINUE:
> + poll_num = poll(fds, 1, 1);
> + if (poll_num == -1) {
> + if (errno == EINTR)
> + continue;
> + error_errno(_("polling inotify message stream failed"));
> + state->listen_data->shutdown = SHUTDOWN_ERROR;
> + continue;
> + }
> +
> + if ((time(NULL) - checked) >= interval) {
> + checked = time(NULL);
As this is linux-specific code, shouldn't we use the linux-specific API
to get the guaranteed atomically growing time here, arther than time()?
Maybe not, but I wonder if this has funny side-effects with ntpd
adjusting the time concurrently...
^ permalink raw reply [flat|nested] 89+ messages in thread
* [PATCH v4 4/6] fsmonitor: enable fsmonitor for Linux
2022-11-23 19:00 ` [PATCH v4 " Eric DeCosta via GitGitGadget
` (2 preceding siblings ...)
2022-11-23 19:00 ` [PATCH v4 3/6] fsmonitor: implement filesystem change listener for Linux Eric DeCosta via GitGitGadget
@ 2022-11-23 19:00 ` Eric DeCosta via GitGitGadget
2022-11-23 19:00 ` [PATCH v4 5/6] fsmonitor: test updates Eric DeCosta via GitGitGadget
` (2 subsequent siblings)
6 siblings, 0 replies; 89+ messages in thread
From: Eric DeCosta via GitGitGadget @ 2022-11-23 19:00 UTC (permalink / raw)
To: git
Cc: Eric Sunshine, Ævar Arnfjörð Bjarmason, Glen Choo,
Johannes Schindelin, Taylor Blau, Eric DeCosta, Eric DeCosta
From: Eric DeCosta <edecosta@mathworks.com>
Update build to enable fsmonitor for Linux.
Signed-off-by: Eric DeCosta <edecosta@mathworks.com>
---
config.mak.uname | 8 ++++++++
contrib/buildsystems/CMakeLists.txt | 11 ++++++++++-
2 files changed, 18 insertions(+), 1 deletion(-)
diff --git a/config.mak.uname b/config.mak.uname
index d63629fe807..5890d810463 100644
--- a/config.mak.uname
+++ b/config.mak.uname
@@ -68,6 +68,14 @@ ifeq ($(uname_S),Linux)
ifneq ($(findstring .el7.,$(uname_R)),)
BASIC_CFLAGS += -std=c99
endif
+ # The builtin FSMonitor on Linux builds upon Simple-IPC. Both require
+ # Unix domain sockets and PThreads.
+ ifndef NO_PTHREADS
+ ifndef NO_UNIX_SOCKETS
+ FSMONITOR_DAEMON_BACKEND = linux
+ FSMONITOR_OS_SETTINGS = linux
+ endif
+ endif
endif
ifeq ($(uname_S),GNU/kFreeBSD)
HAVE_ALLOCA_H = YesPlease
diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt
index 3957e4cf8cd..f058c3d19b4 100644
--- a/contrib/buildsystems/CMakeLists.txt
+++ b/contrib/buildsystems/CMakeLists.txt
@@ -304,7 +304,16 @@ else()
endif()
if(SUPPORTS_SIMPLE_IPC)
- if(CMAKE_SYSTEM_NAME STREQUAL "Windows")
+ if(CMAKE_SYSTEM_NAME STREQUAL "Linux")
+ add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)
+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-linux.c)
+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-linux.c)
+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-linux.c)
+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-linux.c)
+
+ add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)
+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-linux.c)
+ elseif(CMAKE_SYSTEM_NAME STREQUAL "Windows")
add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)
list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)
list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-win32.c)
--
gitgitgadget
^ permalink raw reply related [flat|nested] 89+ messages in thread* [PATCH v4 5/6] fsmonitor: test updates
2022-11-23 19:00 ` [PATCH v4 " Eric DeCosta via GitGitGadget
` (3 preceding siblings ...)
2022-11-23 19:00 ` [PATCH v4 4/6] fsmonitor: enable fsmonitor " Eric DeCosta via GitGitGadget
@ 2022-11-23 19:00 ` Eric DeCosta via GitGitGadget
2022-11-23 19:00 ` [PATCH v4 6/6] fsmonitor: update doc for Linux Eric DeCosta via GitGitGadget
2022-12-12 21:57 ` [PATCH v5 0/6] fsmonitor: Implement fsmonitor " Eric DeCosta via GitGitGadget
6 siblings, 0 replies; 89+ messages in thread
From: Eric DeCosta via GitGitGadget @ 2022-11-23 19:00 UTC (permalink / raw)
To: git
Cc: Eric Sunshine, Ævar Arnfjörð Bjarmason, Glen Choo,
Johannes Schindelin, Taylor Blau, Eric DeCosta, Eric DeCosta
From: Eric DeCosta <edecosta@mathworks.com>
t7527-builtin-fsmonitor was leaking fsmonitor--daemon processes in some
cases.
Accomodate slight difference in the number of events generated on Linux.
On lower-powered systems, spin a little to give the daemon time
to respond to and log filesystem events.
Signed-off-by: Eric DeCosta <edecosta@mathworks.com>
---
t/t7527-builtin-fsmonitor.sh | 94 ++++++++++++++++++++++++++++++------
1 file changed, 80 insertions(+), 14 deletions(-)
diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh
index 4abc74db2bb..951374231b7 100755
--- a/t/t7527-builtin-fsmonitor.sh
+++ b/t/t7527-builtin-fsmonitor.sh
@@ -13,7 +13,7 @@ fi
stop_daemon_delete_repo () {
r=$1 &&
test_might_fail git -C $r fsmonitor--daemon stop &&
- rm -rf $1
+ rm -rf $r
}
start_daemon () {
@@ -72,6 +72,34 @@ start_daemon () {
)
}
+IMPLICIT_TIMEOUT=5
+
+wait_for_update () {
+ func=$1 &&
+ file=$2 &&
+ sz=$(wc -c < "$file") &&
+ last=0 &&
+ $func &&
+ k=0 &&
+ while test "$k" -lt $IMPLICIT_TIMEOUT
+ do
+ nsz=$(wc -c < "$file")
+ if test "$nsz" -gt "$sz"
+ then
+ if test "$last" -eq "$nsz"
+ then
+ cat "$file" &&
+ return 0
+ fi
+ last=$nsz
+ fi
+ sleep 1
+ k=$(( $k + 1 ))
+ done &&
+ cat "$file" &&
+ return 0
+}
+
# Is a Trace2 data event present with the given catetory and key?
# We do not care what the value is.
#
@@ -137,7 +165,6 @@ test_expect_success 'implicit daemon start' '
# machines (where it might take a moment to wake and reschedule the
# daemon process) to avoid false alarms during test runs.)
#
-IMPLICIT_TIMEOUT=5
verify_implicit_shutdown () {
r=$1 &&
@@ -373,6 +400,15 @@ create_files () {
echo 3 >dir2/new
}
+rename_directory () {
+ mv dirtorename dirrenamed
+}
+
+rename_directory_file () {
+ mv dirtorename dirrenamed &&
+ echo 1 > dirrenamed/new
+}
+
rename_files () {
mv rename renamed &&
mv dir1/rename dir1/renamed &&
@@ -427,10 +463,12 @@ test_expect_success 'edit some files' '
start_daemon --tf "$PWD/.git/trace" &&
- edit_files &&
+ wait_for_update edit_files "$PWD/.git/trace" &&
test-tool fsmonitor-client query --token 0 &&
+ test_might_fail git fsmonitor--daemon stop &&
+
grep "^event: dir1/modified$" .git/trace &&
grep "^event: dir2/modified$" .git/trace &&
grep "^event: modified$" .git/trace &&
@@ -442,10 +480,12 @@ test_expect_success 'create some files' '
start_daemon --tf "$PWD/.git/trace" &&
- create_files &&
+ wait_for_update create_files "$PWD/.git/trace" &&
test-tool fsmonitor-client query --token 0 &&
+ test_might_fail git fsmonitor--daemon stop &&
+
grep "^event: dir1/new$" .git/trace &&
grep "^event: dir2/new$" .git/trace &&
grep "^event: new$" .git/trace
@@ -456,10 +496,12 @@ test_expect_success 'delete some files' '
start_daemon --tf "$PWD/.git/trace" &&
- delete_files &&
+ wait_for_update delete_files "$PWD/.git/trace" &&
test-tool fsmonitor-client query --token 0 &&
+ test_might_fail git fsmonitor--daemon stop &&
+
grep "^event: dir1/delete$" .git/trace &&
grep "^event: dir2/delete$" .git/trace &&
grep "^event: delete$" .git/trace
@@ -470,10 +512,12 @@ test_expect_success 'rename some files' '
start_daemon --tf "$PWD/.git/trace" &&
- rename_files &&
+ wait_for_update rename_files "$PWD/.git/trace" &&
test-tool fsmonitor-client query --token 0 &&
+ test_might_fail git fsmonitor--daemon stop &&
+
grep "^event: dir1/rename$" .git/trace &&
grep "^event: dir2/rename$" .git/trace &&
grep "^event: rename$" .git/trace &&
@@ -487,23 +531,42 @@ test_expect_success 'rename directory' '
start_daemon --tf "$PWD/.git/trace" &&
- mv dirtorename dirrenamed &&
+ wait_for_update rename_directory "$PWD/.git/trace" &&
test-tool fsmonitor-client query --token 0 &&
+ test_might_fail git fsmonitor--daemon stop &&
+
grep "^event: dirtorename/*$" .git/trace &&
grep "^event: dirrenamed/*$" .git/trace
'
+test_expect_success 'rename directory file' '
+ test_when_finished clean_up_repo_and_stop_daemon &&
+
+ start_daemon --tf "$PWD/.git/trace" &&
+
+ wait_for_update rename_directory_file "$PWD/.git/trace" &&
+
+ test-tool fsmonitor-client query --token 0 &&
+
+ test_might_fail git fsmonitor--daemon stop &&
+
+ grep "^event: dirtorename/*$" .git/trace &&
+ grep "^event: dirrenamed/*$" .git/trace &&
+ grep "^event: dirrenamed/new$" .git/trace
+'
test_expect_success 'file changes to directory' '
test_when_finished clean_up_repo_and_stop_daemon &&
start_daemon --tf "$PWD/.git/trace" &&
- file_to_directory &&
+ wait_for_update file_to_directory "$PWD/.git/trace" &&
test-tool fsmonitor-client query --token 0 &&
+ test_might_fail git fsmonitor--daemon stop &&
+
grep "^event: delete$" .git/trace &&
grep "^event: delete/new$" .git/trace
'
@@ -513,10 +576,12 @@ test_expect_success 'directory changes to a file' '
start_daemon --tf "$PWD/.git/trace" &&
- directory_to_file &&
+ wait_for_update directory_to_file "$PWD/.git/trace" &&
test-tool fsmonitor-client query --token 0 &&
+ test_might_fail git fsmonitor--daemon stop &&
+
grep "^event: dir1$" .git/trace
'
@@ -561,7 +626,7 @@ test_expect_success 'flush cached data' '
test-tool -C test_flush fsmonitor-client query --token "builtin:test_00000002:0" >actual_2 &&
nul_to_q <actual_2 >actual_q2 &&
- grep "^builtin:test_00000002:0Q$" actual_q2 &&
+ grep "^builtin:test_00000002:[0-1]Q$" actual_q2 &&
>test_flush/file_3 &&
@@ -732,7 +797,8 @@ u_values="$u1 $u2"
for u in $u_values
do
test_expect_success "unicode in repo root path: $u" '
- test_when_finished "stop_daemon_delete_repo $u" &&
+ test_when_finished \
+ "stop_daemon_delete_repo `echo "$u" | sed 's:x:\\\\\\\\\\\\\\x:g'`" &&
git init "$u" &&
echo 1 >"$u"/file1 &&
@@ -818,8 +884,7 @@ test_expect_success 'submodule setup' '
'
test_expect_success 'submodule always visited' '
- test_when_finished "git -C super fsmonitor--daemon stop; \
- rm -rf super; \
+ test_when_finished "rm -rf super; \
rm -rf sub" &&
create_super super &&
@@ -887,7 +952,8 @@ have_t2_error_event () {
}
test_expect_success "stray submodule super-prefix warning" '
- test_when_finished "rm -rf super; \
+ test_when_finished "git -C super/dir_1/dir_2/sub fsmonitor--daemon stop; \
+ rm -rf super; \
rm -rf sub; \
rm super-sub.trace" &&
--
gitgitgadget
^ permalink raw reply related [flat|nested] 89+ messages in thread* [PATCH v4 6/6] fsmonitor: update doc for Linux
2022-11-23 19:00 ` [PATCH v4 " Eric DeCosta via GitGitGadget
` (4 preceding siblings ...)
2022-11-23 19:00 ` [PATCH v4 5/6] fsmonitor: test updates Eric DeCosta via GitGitGadget
@ 2022-11-23 19:00 ` Eric DeCosta via GitGitGadget
2022-12-12 21:57 ` [PATCH v5 0/6] fsmonitor: Implement fsmonitor " Eric DeCosta via GitGitGadget
6 siblings, 0 replies; 89+ messages in thread
From: Eric DeCosta via GitGitGadget @ 2022-11-23 19:00 UTC (permalink / raw)
To: git
Cc: Eric Sunshine, Ævar Arnfjörð Bjarmason, Glen Choo,
Johannes Schindelin, Taylor Blau, Eric DeCosta, Eric DeCosta
From: Eric DeCosta <edecosta@mathworks.com>
Update the documentation for Linux.
Signed-off-by: Eric DeCosta <edecosta@mathworks.com>
---
Documentation/config/fsmonitor--daemon.txt | 4 ++--
Documentation/git-fsmonitor--daemon.txt | 24 ++++++++++++++--------
2 files changed, 18 insertions(+), 10 deletions(-)
diff --git a/Documentation/config/fsmonitor--daemon.txt b/Documentation/config/fsmonitor--daemon.txt
index c225c6c9e74..2cafb040d96 100644
--- a/Documentation/config/fsmonitor--daemon.txt
+++ b/Documentation/config/fsmonitor--daemon.txt
@@ -4,8 +4,8 @@ fsmonitor.allowRemote::
behavior. Only respected when `core.fsmonitor` is set to `true`.
fsmonitor.socketDir::
- This Mac OS-specific option, if set, specifies the directory in
+ Mac OS and Linux-specific option. If set, specifies the directory in
which to create the Unix domain socket used for communication
between the fsmonitor daemon and various Git commands. The directory must
- reside on a native Mac OS filesystem. Only respected when `core.fsmonitor`
+ reside on a native filesystem. Only respected when `core.fsmonitor`
is set to `true`.
diff --git a/Documentation/git-fsmonitor--daemon.txt b/Documentation/git-fsmonitor--daemon.txt
index 8238eadb0e1..c2b08229c74 100644
--- a/Documentation/git-fsmonitor--daemon.txt
+++ b/Documentation/git-fsmonitor--daemon.txt
@@ -76,23 +76,31 @@ repositories; this may be overridden by setting `fsmonitor.allowRemote` to
correctly with all network-mounted repositories and such use is considered
experimental.
-On Mac OS, the inter-process communication (IPC) between various Git
+On Linux and Mac OS, the inter-process communication (IPC) between various Git
commands and the fsmonitor daemon is done via a Unix domain socket (UDS) -- a
-special type of file -- which is supported by native Mac OS filesystems,
-but not on network-mounted filesystems, NTFS, or FAT32. Other filesystems
-may or may not have the needed support; the fsmonitor daemon is not guaranteed
-to work with these filesystems and such use is considered experimental.
+special type of file -- which is supported by many native Linux and Mac OS
+filesystems, but not on network-mounted filesystems, NTFS, or FAT32. Other
+filesystems may or may not have the needed support; the fsmonitor daemon is not
+guaranteed to work with these filesystems and such use is considered
+experimental.
By default, the socket is created in the `.git` directory, however, if the
`.git` directory is on a network-mounted filesystem, it will be instead be
created at `$HOME/.git-fsmonitor-*` unless `$HOME` itself is on a
network-mounted filesystem in which case you must set the configuration
-variable `fsmonitor.socketDir` to the path of a directory on a Mac OS native
+variable `fsmonitor.socketDir` to the path of a directory on a native
filesystem in which to create the socket file.
If none of the above directories (`.git`, `$HOME`, or `fsmonitor.socketDir`)
-is on a native Mac OS file filesystem the fsmonitor daemon will report an
-error that will cause the daemon and the currently running command to exit.
+is on a native Linux or Mac OS filesystem the fsmonitor daemon will report
+an error that will cause the daemon to exit and the currently running command
+to issue a warning.
+
+On Linux, the fsmonitor daemon registers a watch for each directory in the
+repository. The default per-user limit for the number of watches on most Linux
+systems is 8192. This may not be sufficient for large repositories or if
+multiple instances of the fsmonitor daemon are running.
+See https://watchexec.github.io/docs/inotify-limits.html[Linux inotify limits] for more information.
CONFIGURATION
-------------
--
gitgitgadget
^ permalink raw reply related [flat|nested] 89+ messages in thread* [PATCH v5 0/6] fsmonitor: Implement fsmonitor for Linux
2022-11-23 19:00 ` [PATCH v4 " Eric DeCosta via GitGitGadget
` (5 preceding siblings ...)
2022-11-23 19:00 ` [PATCH v4 6/6] fsmonitor: update doc for Linux Eric DeCosta via GitGitGadget
@ 2022-12-12 21:57 ` Eric DeCosta via GitGitGadget
2022-12-12 21:58 ` [PATCH v5 1/6] fsmonitor: prepare to share code between Mac OS and Linux Eric DeCosta via GitGitGadget
` (6 more replies)
6 siblings, 7 replies; 89+ messages in thread
From: Eric DeCosta via GitGitGadget @ 2022-12-12 21:57 UTC (permalink / raw)
To: git
Cc: Eric Sunshine, Ævar Arnfjörð Bjarmason, Glen Choo,
Johannes Schindelin, Taylor Blau, Eric DeCosta
Goal is to deliver fsmonitor for Linux that is on par with fsmonitor for
Windows and Mac OS.
This patch set builds upon previous work for done for Windows and Mac OS to
implement a fsmonitor back-end for Linux based on the Linux inotify API.
inotify differs significantly from the equivalent Windows and Mac OS APIs in
that a watch must be registered for every directory of interest (rather than
a singular watch at the root of the directory tree) and special care must be
taken to handle directory renames correctly.
More information about inotify:
https://man7.org/linux/man-pages/man7/inotify.7.html
v4 differs from v3:
* Code review feedback
v3 differs from v2:
* Avoid potential entanglements with GPLv3
* Classify a reasonable set of filesystems as being remote
v2 differs from v1:
* Prior work for Windows and Mac OS has been merged to master, reducing the
patch set from 12 to 6 patches
* Code review feedback
* Identified and resolved race condition revealed by CI test system, see
"Limitations and caveats" regarding monitoring of directory trees from
the man page, above
* Apologies for being away from this for so long, but my attention was
needed elsewhere
v1 differs from v0:
* Code review feedback
* Update how and which code can be shared between Mac OS and Linux
* Increase polling frequency to every 1ms (matches Mac OS)
* Updates to t7527 to improve test stability
Eric DeCosta (6):
fsmonitor: prepare to share code between Mac OS and Linux
fsmonitor: determine if filesystem is local or remote
fsmonitor: implement filesystem change listener for Linux
fsmonitor: enable fsmonitor for Linux
fsmonitor: test updates
fsmonitor: update doc for Linux
Documentation/config/fsmonitor--daemon.txt | 4 +-
Documentation/git-fsmonitor--daemon.txt | 24 +-
compat/fsmonitor/fsm-health-linux.c | 24 +
compat/fsmonitor/fsm-ipc-darwin.c | 53 +-
compat/fsmonitor/fsm-ipc-linux.c | 1 +
compat/fsmonitor/fsm-ipc-unix.c | 52 ++
compat/fsmonitor/fsm-listen-linux.c | 676 +++++++++++++++++++++
compat/fsmonitor/fsm-path-utils-linux.c | 193 ++++++
compat/fsmonitor/fsm-settings-darwin.c | 63 +-
compat/fsmonitor/fsm-settings-linux.c | 1 +
compat/fsmonitor/fsm-settings-unix.c | 62 ++
config.mak.uname | 8 +
contrib/buildsystems/CMakeLists.txt | 11 +-
t/t7527-builtin-fsmonitor.sh | 94 ++-
14 files changed, 1127 insertions(+), 139 deletions(-)
create mode 100644 compat/fsmonitor/fsm-health-linux.c
create mode 100644 compat/fsmonitor/fsm-ipc-linux.c
create mode 100644 compat/fsmonitor/fsm-ipc-unix.c
create mode 100644 compat/fsmonitor/fsm-listen-linux.c
create mode 100644 compat/fsmonitor/fsm-path-utils-linux.c
create mode 100644 compat/fsmonitor/fsm-settings-linux.c
create mode 100644 compat/fsmonitor/fsm-settings-unix.c
base-commit: c48035d29b4e524aed3a32f0403676f0d9128863
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1352%2Fedecosta-mw%2Ffsmonitor_linux-v5
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1352/edecosta-mw/fsmonitor_linux-v5
Pull-Request: https://github.com/git/git/pull/1352
Range-diff vs v4:
1: 99d684c7bdf ! 1: 7f659603c9a fsmonitor: prepare to share code between Mac OS and Linux
@@ compat/fsmonitor/fsm-settings-unix.c (new)
+ strbuf_addstr(&path, ipc_path);
+
+ if (fsmonitor__get_fs_info(dirname(path.buf), &fs) == -1) {
++ free(fs.typename);
+ strbuf_release(&path);
+ return FSMONITOR_REASON_ERROR;
+ }
2: e53fc077540 ! 2: eb3ff9d1c05 fsmonitor: determine if filesystem is local or remote
@@ compat/fsmonitor/fsm-path-utils-linux.c (new)
+#include <sys/vfs.h>
+#include <sys/statvfs.h>
+
-+static int is_remote_fs(const char* path) {
++static int is_remote_fs(const char *path) {
+ struct statfs fs;
+
+ if (statfs(path, &fs)) {
@@ compat/fsmonitor/fsm-path-utils-linux.c (new)
+ }
+
+ switch (fs.f_type) {
-+ case 0x61636673: /* ACFS */
-+ case 0x5346414F: /* AFS */
-+ case 0x00C36400: /* CEPH */
-+ case 0xFF534D42: /* CIFS */
-+ case 0x73757245: /* CODA */
-+ case 0x19830326: /* FHGFS */
-+ case 0x1161970: /* GFS */
-+ case 0x47504653: /* GPFS */
-+ case 0x013111A8: /* IBRIX */
-+ case 0x6B414653: /* KAFS */
-+ case 0x0BD00BD0: /* LUSTRE */
-+ case 0x564C: /* NCP */
-+ case 0x6969: /* NFS */
-+ case 0x6E667364: /* NFSD */
-+ case 0x7461636f: /* OCFS2 */
-+ case 0xAAD7AAEA: /* PANFS */
-+ case 0x517B: /* SMB */
-+ case 0xBEEFDEAD: /* SNFS */
-+ case 0xFE534D42: /* SMB2 */
-+ case 0xBACBACBC: /* VMHGFS */
-+ case 0xA501FCF5: /* VXFS */
-+ return 1;
-+ default:
-+ break;
++ case 0x61636673: /* ACFS */
++ case 0x5346414F: /* AFS */
++ case 0x00C36400: /* CEPH */
++ case 0xFF534D42: /* CIFS */
++ case 0x73757245: /* CODA */
++ case 0x19830326: /* FHGFS */
++ case 0x1161970: /* GFS */
++ case 0x47504653: /* GPFS */
++ case 0x013111A8: /* IBRIX */
++ case 0x6B414653: /* KAFS */
++ case 0x0BD00BD0: /* LUSTRE */
++ case 0x564C: /* NCP */
++ case 0x6969: /* NFS */
++ case 0x6E667364: /* NFSD */
++ case 0x7461636f: /* OCFS2 */
++ case 0xAAD7AAEA: /* PANFS */
++ case 0x517B: /* SMB */
++ case 0xBEEFDEAD: /* SNFS */
++ case 0xFE534D42: /* SMB2 */
++ case 0xBACBACBC: /* VMHGFS */
++ case 0xA501FCF5: /* VXFS */
++ return 1;
++ default:
++ break;
+ }
+
+ return 0;
+}
+
+static int find_mount(const char *path, const struct statvfs *fs,
-+ struct mntent *ent)
++ struct mntent *entry)
+{
+ const char *const mounts = "/proc/mounts";
-+ const char *rp = real_pathdup(path, 1);
++ char *rp = real_pathdup(path, 1);
+ struct mntent *ment = NULL;
+ struct statvfs mntfs;
+ FILE *fp;
+ int found = 0;
+ int dlen, plen, flen = 0;
+
-+ ent->mnt_fsname = NULL;
-+ ent->mnt_dir = NULL;
-+ ent->mnt_type = NULL;
++ entry->mnt_fsname = NULL;
++ entry->mnt_dir = NULL;
++ entry->mnt_type = NULL;
+
+ fp = setmntent(mounts, "r");
+ if (!fp) {
++ free(rp);
+ error_errno(_("setmntent('%s') failed"), mounts);
+ return -1;
+ }
@@ compat/fsmonitor/fsm-path-utils-linux.c (new)
+ default:
+ error_errno(_("statvfs('%s') failed"), ment->mnt_dir);
+ endmntent(fp);
++ free(rp);
+ return -1;
+ }
+ }
@@ compat/fsmonitor/fsm-path-utils-linux.c (new)
+ if (dlen > plen)
+ continue;
+ /*
-+ * root is always a potential match; otherwise look for
-+ * directory prefix
++ * look for the longest prefix (including root)
+ */
-+ if ((dlen == 1 && ment->mnt_dir[0] == '/') ||
-+ (dlen > flen && (!rp[dlen] || rp[dlen] == '/'))) {
++ if (dlen > flen &&
++ ((dlen == 1 && ment->mnt_dir[0] == '/') ||
++ (!rp[dlen] || rp[dlen] == '/'))) {
+ flen = dlen;
++ found = 1;
++
+ /*
+ * https://man7.org/linux/man-pages/man3/getmntent.3.html
+ *
+ * The pointer points to a static area of memory which is
+ * overwritten by subsequent calls to getmntent().
+ */
-+ found = 1;
-+ free(ent->mnt_fsname);
-+ free(ent->mnt_dir);
-+ free(ent->mnt_type);
-+ ent->mnt_fsname = xstrdup(ment->mnt_fsname);
-+ ent->mnt_dir = xstrdup(ment->mnt_dir);
-+ ent->mnt_type = xstrdup(ment->mnt_type);
++ free(entry->mnt_fsname);
++ free(entry->mnt_dir);
++ free(entry->mnt_type);
++ entry->mnt_fsname = xstrdup(ment->mnt_fsname);
++ entry->mnt_dir = xstrdup(ment->mnt_dir);
++ entry->mnt_type = xstrdup(ment->mnt_type);
+ }
+ }
+ }
+ endmntent(fp);
++ free(rp);
+
+ if (!found)
+ return -1;
@@ compat/fsmonitor/fsm-path-utils-linux.c (new)
+
+int fsmonitor__get_fs_info(const char *path, struct fs_info *fs_info)
+{
-+ struct mntent ment;
++ struct mntent entry;
+ struct statvfs fs;
+
++ fs_info->is_remote = -1;
++ fs_info->typename = NULL;
++
+ if (statvfs(path, &fs))
+ return error_errno(_("statvfs('%s') failed"), path);
+
-+
-+ if (find_mount(path, &fs, &ment) < 0) {
-+ free(ment.mnt_fsname);
-+ free(ment.mnt_dir);
-+ free(ment.mnt_type);
++ if (find_mount(path, &fs, &entry) < 0) {
++ free(entry.mnt_fsname);
++ free(entry.mnt_dir);
++ free(entry.mnt_type);
+ return -1;
+ }
+
+ trace_printf_key(&trace_fsmonitor,
+ "statvfs('%s') [flags 0x%08lx] '%s' '%s'",
-+ path, fs.f_flag, ment.mnt_type, ment.mnt_fsname);
++ path, fs.f_flag, entry.mnt_type, entry.mnt_fsname);
+
-+ fs_info->is_remote = is_remote_fs(ment.mnt_dir);
-+ fs_info->typename = ment.mnt_fsname;
-+ free(ment.mnt_dir);
-+ free(ment.mnt_type);
++ fs_info->is_remote = is_remote_fs(entry.mnt_dir);
++ fs_info->typename = xstrdup(entry.mnt_fsname);
++ free(entry.mnt_fsname);
++ free(entry.mnt_dir);
++ free(entry.mnt_type);
+
-+ if (fs_info->is_remote < 0) {
-+ free(ment.mnt_fsname);
++ if (fs_info->is_remote < 0)
+ return -1;
-+ }
+
+ trace_printf_key(&trace_fsmonitor,
+ "'%s' is_remote: %d",
@@ compat/fsmonitor/fsm-path-utils-linux.c (new)
+{
+ struct fs_info fs;
+
-+ if (fsmonitor__get_fs_info(path, &fs))
++ if (fsmonitor__get_fs_info(path, &fs)) {
++ free(fs.typename);
+ return -1;
++ }
+
+ free(fs.typename);
+
3: 80282efef57 = 3: e093a2703b1 fsmonitor: implement filesystem change listener for Linux
4: cb03803e355 = 4: c03070fb0a2 fsmonitor: enable fsmonitor for Linux
5: 8d9d469b356 = 5: 6a7b554642c fsmonitor: test updates
6: 5afd03fa6ca = 6: 827410c22ee fsmonitor: update doc for Linux
--
gitgitgadget
^ permalink raw reply [flat|nested] 89+ messages in thread* [PATCH v5 1/6] fsmonitor: prepare to share code between Mac OS and Linux
2022-12-12 21:57 ` [PATCH v5 0/6] fsmonitor: Implement fsmonitor " Eric DeCosta via GitGitGadget
@ 2022-12-12 21:58 ` Eric DeCosta via GitGitGadget
2022-12-12 21:58 ` [PATCH v5 2/6] fsmonitor: determine if filesystem is local or remote Eric DeCosta via GitGitGadget
` (5 subsequent siblings)
6 siblings, 0 replies; 89+ messages in thread
From: Eric DeCosta via GitGitGadget @ 2022-12-12 21:58 UTC (permalink / raw)
To: git
Cc: Eric Sunshine, Ævar Arnfjörð Bjarmason, Glen Choo,
Johannes Schindelin, Taylor Blau, Eric DeCosta, Eric DeCosta
From: Eric DeCosta <edecosta@mathworks.com>
Linux and Mac OS can share some of the code originally developed for Mac OS.
Mac OS and Linux can share fsm-ipc-unix.c and fsm-settings-unix.c
Signed-off-by: Eric DeCosta <edecosta@mathworks.com>
---
compat/fsmonitor/fsm-health-linux.c | 24 ++++++++++
compat/fsmonitor/fsm-ipc-darwin.c | 53 +---------------------
compat/fsmonitor/fsm-ipc-linux.c | 1 +
compat/fsmonitor/fsm-ipc-unix.c | 52 +++++++++++++++++++++
compat/fsmonitor/fsm-settings-darwin.c | 63 +-------------------------
compat/fsmonitor/fsm-settings-linux.c | 1 +
compat/fsmonitor/fsm-settings-unix.c | 62 +++++++++++++++++++++++++
7 files changed, 142 insertions(+), 114 deletions(-)
create mode 100644 compat/fsmonitor/fsm-health-linux.c
create mode 100644 compat/fsmonitor/fsm-ipc-linux.c
create mode 100644 compat/fsmonitor/fsm-ipc-unix.c
create mode 100644 compat/fsmonitor/fsm-settings-linux.c
create mode 100644 compat/fsmonitor/fsm-settings-unix.c
diff --git a/compat/fsmonitor/fsm-health-linux.c b/compat/fsmonitor/fsm-health-linux.c
new file mode 100644
index 00000000000..b9f709e8548
--- /dev/null
+++ b/compat/fsmonitor/fsm-health-linux.c
@@ -0,0 +1,24 @@
+#include "cache.h"
+#include "config.h"
+#include "fsmonitor.h"
+#include "fsm-health.h"
+#include "fsmonitor--daemon.h"
+
+int fsm_health__ctor(struct fsmonitor_daemon_state *state)
+{
+ return 0;
+}
+
+void fsm_health__dtor(struct fsmonitor_daemon_state *state)
+{
+ return;
+}
+
+void fsm_health__loop(struct fsmonitor_daemon_state *state)
+{
+ return;
+}
+
+void fsm_health__stop_async(struct fsmonitor_daemon_state *state)
+{
+}
diff --git a/compat/fsmonitor/fsm-ipc-darwin.c b/compat/fsmonitor/fsm-ipc-darwin.c
index d67b0ee50d3..4c3c92081ee 100644
--- a/compat/fsmonitor/fsm-ipc-darwin.c
+++ b/compat/fsmonitor/fsm-ipc-darwin.c
@@ -1,52 +1 @@
-#include "cache.h"
-#include "config.h"
-#include "strbuf.h"
-#include "fsmonitor.h"
-#include "fsmonitor-ipc.h"
-#include "fsmonitor-path-utils.h"
-
-static GIT_PATH_FUNC(fsmonitor_ipc__get_default_path, "fsmonitor--daemon.ipc")
-
-const char *fsmonitor_ipc__get_path(struct repository *r)
-{
- static const char *ipc_path = NULL;
- git_SHA_CTX sha1ctx;
- char *sock_dir = NULL;
- struct strbuf ipc_file = STRBUF_INIT;
- unsigned char hash[GIT_MAX_RAWSZ];
-
- if (!r)
- BUG("No repository passed into fsmonitor_ipc__get_path");
-
- if (ipc_path)
- return ipc_path;
-
-
- /* By default the socket file is created in the .git directory */
- if (fsmonitor__is_fs_remote(r->gitdir) < 1) {
- ipc_path = fsmonitor_ipc__get_default_path();
- return ipc_path;
- }
-
- git_SHA1_Init(&sha1ctx);
- git_SHA1_Update(&sha1ctx, r->worktree, strlen(r->worktree));
- git_SHA1_Final(hash, &sha1ctx);
-
- repo_config_get_string(r, "fsmonitor.socketdir", &sock_dir);
-
- /* Create the socket file in either socketDir or $HOME */
- if (sock_dir && *sock_dir) {
- strbuf_addf(&ipc_file, "%s/.git-fsmonitor-%s",
- sock_dir, hash_to_hex(hash));
- } else {
- strbuf_addf(&ipc_file, "~/.git-fsmonitor-%s", hash_to_hex(hash));
- }
- free(sock_dir);
-
- ipc_path = interpolate_path(ipc_file.buf, 1);
- if (!ipc_path)
- die(_("Invalid path: %s"), ipc_file.buf);
-
- strbuf_release(&ipc_file);
- return ipc_path;
-}
+#include "fsm-ipc-unix.c"
diff --git a/compat/fsmonitor/fsm-ipc-linux.c b/compat/fsmonitor/fsm-ipc-linux.c
new file mode 100644
index 00000000000..4c3c92081ee
--- /dev/null
+++ b/compat/fsmonitor/fsm-ipc-linux.c
@@ -0,0 +1 @@
+#include "fsm-ipc-unix.c"
diff --git a/compat/fsmonitor/fsm-ipc-unix.c b/compat/fsmonitor/fsm-ipc-unix.c
new file mode 100644
index 00000000000..d67b0ee50d3
--- /dev/null
+++ b/compat/fsmonitor/fsm-ipc-unix.c
@@ -0,0 +1,52 @@
+#include "cache.h"
+#include "config.h"
+#include "strbuf.h"
+#include "fsmonitor.h"
+#include "fsmonitor-ipc.h"
+#include "fsmonitor-path-utils.h"
+
+static GIT_PATH_FUNC(fsmonitor_ipc__get_default_path, "fsmonitor--daemon.ipc")
+
+const char *fsmonitor_ipc__get_path(struct repository *r)
+{
+ static const char *ipc_path = NULL;
+ git_SHA_CTX sha1ctx;
+ char *sock_dir = NULL;
+ struct strbuf ipc_file = STRBUF_INIT;
+ unsigned char hash[GIT_MAX_RAWSZ];
+
+ if (!r)
+ BUG("No repository passed into fsmonitor_ipc__get_path");
+
+ if (ipc_path)
+ return ipc_path;
+
+
+ /* By default the socket file is created in the .git directory */
+ if (fsmonitor__is_fs_remote(r->gitdir) < 1) {
+ ipc_path = fsmonitor_ipc__get_default_path();
+ return ipc_path;
+ }
+
+ git_SHA1_Init(&sha1ctx);
+ git_SHA1_Update(&sha1ctx, r->worktree, strlen(r->worktree));
+ git_SHA1_Final(hash, &sha1ctx);
+
+ repo_config_get_string(r, "fsmonitor.socketdir", &sock_dir);
+
+ /* Create the socket file in either socketDir or $HOME */
+ if (sock_dir && *sock_dir) {
+ strbuf_addf(&ipc_file, "%s/.git-fsmonitor-%s",
+ sock_dir, hash_to_hex(hash));
+ } else {
+ strbuf_addf(&ipc_file, "~/.git-fsmonitor-%s", hash_to_hex(hash));
+ }
+ free(sock_dir);
+
+ ipc_path = interpolate_path(ipc_file.buf, 1);
+ if (!ipc_path)
+ die(_("Invalid path: %s"), ipc_file.buf);
+
+ strbuf_release(&ipc_file);
+ return ipc_path;
+}
diff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c
index 6abbc7af3ab..14baf9f0603 100644
--- a/compat/fsmonitor/fsm-settings-darwin.c
+++ b/compat/fsmonitor/fsm-settings-darwin.c
@@ -1,62 +1 @@
-#include "config.h"
-#include "fsmonitor.h"
-#include "fsmonitor-ipc.h"
-#include "fsmonitor-settings.h"
-#include "fsmonitor-path-utils.h"
-
- /*
- * For the builtin FSMonitor, we create the Unix domain socket for the
- * IPC in the .git directory. If the working directory is remote,
- * then the socket will be created on the remote file system. This
- * can fail if the remote file system does not support UDS file types
- * (e.g. smbfs to a Windows server) or if the remote kernel does not
- * allow a non-local process to bind() the socket. (These problems
- * could be fixed by moving the UDS out of the .git directory and to a
- * well-known local directory on the client machine, but care should
- * be taken to ensure that $HOME is actually local and not a managed
- * file share.)
- *
- * FAT32 and NTFS working directories are problematic too.
- *
- * The builtin FSMonitor uses a Unix domain socket in the .git
- * directory for IPC. These Windows drive formats do not support
- * Unix domain sockets, so mark them as incompatible for the daemon.
- *
- */
-static enum fsmonitor_reason check_uds_volume(struct repository *r)
-{
- struct fs_info fs;
- const char *ipc_path = fsmonitor_ipc__get_path(r);
- struct strbuf path = STRBUF_INIT;
- strbuf_add(&path, ipc_path, strlen(ipc_path));
-
- if (fsmonitor__get_fs_info(dirname(path.buf), &fs) == -1) {
- strbuf_release(&path);
- return FSMONITOR_REASON_ERROR;
- }
-
- strbuf_release(&path);
-
- if (fs.is_remote ||
- !strcmp(fs.typename, "msdos") ||
- !strcmp(fs.typename, "ntfs")) {
- free(fs.typename);
- return FSMONITOR_REASON_NOSOCKETS;
- }
-
- free(fs.typename);
- return FSMONITOR_REASON_OK;
-}
-
-enum fsmonitor_reason fsm_os__incompatible(struct repository *r, int ipc)
-{
- enum fsmonitor_reason reason;
-
- if (ipc) {
- reason = check_uds_volume(r);
- if (reason != FSMONITOR_REASON_OK)
- return reason;
- }
-
- return FSMONITOR_REASON_OK;
-}
+#include "fsm-settings-unix.c"
diff --git a/compat/fsmonitor/fsm-settings-linux.c b/compat/fsmonitor/fsm-settings-linux.c
new file mode 100644
index 00000000000..14baf9f0603
--- /dev/null
+++ b/compat/fsmonitor/fsm-settings-linux.c
@@ -0,0 +1 @@
+#include "fsm-settings-unix.c"
diff --git a/compat/fsmonitor/fsm-settings-unix.c b/compat/fsmonitor/fsm-settings-unix.c
new file mode 100644
index 00000000000..da23254be40
--- /dev/null
+++ b/compat/fsmonitor/fsm-settings-unix.c
@@ -0,0 +1,62 @@
+#include "config.h"
+#include "fsmonitor.h"
+#include "fsmonitor-ipc.h"
+#include "fsmonitor-path-utils.h"
+
+ /*
+ * For the builtin FSMonitor, we create the Unix domain socket for the
+ * IPC in the .git directory. If the working directory is remote,
+ * then the socket will be created on the remote file system. This
+ * can fail if the remote file system does not support UDS file types
+ * (e.g. smbfs to a Windows server) or if the remote kernel does not
+ * allow a non-local process to bind() the socket. (These problems
+ * could be fixed by moving the UDS out of the .git directory and to a
+ * well-known local directory on the client machine, but care should
+ * be taken to ensure that $HOME is actually local and not a managed
+ * file share.)
+ *
+ * FAT32 and NTFS working directories are problematic too.
+ *
+ * The builtin FSMonitor uses a Unix domain socket in the .git
+ * directory for IPC. These Windows drive formats do not support
+ * Unix domain sockets, so mark them as incompatible for the daemon.
+ *
+ */
+static enum fsmonitor_reason check_uds_volume(struct repository *r)
+{
+ struct fs_info fs;
+ const char *ipc_path = fsmonitor_ipc__get_path(r);
+ struct strbuf path = STRBUF_INIT;
+ strbuf_addstr(&path, ipc_path);
+
+ if (fsmonitor__get_fs_info(dirname(path.buf), &fs) == -1) {
+ free(fs.typename);
+ strbuf_release(&path);
+ return FSMONITOR_REASON_ERROR;
+ }
+
+ strbuf_release(&path);
+
+ if (fs.is_remote ||
+ !strcmp(fs.typename, "msdos") ||
+ !strcmp(fs.typename, "ntfs")) {
+ free(fs.typename);
+ return FSMONITOR_REASON_NOSOCKETS;
+ }
+
+ free(fs.typename);
+ return FSMONITOR_REASON_OK;
+}
+
+enum fsmonitor_reason fsm_os__incompatible(struct repository *r, int ipc)
+{
+ enum fsmonitor_reason reason;
+
+ if (ipc) {
+ reason = check_uds_volume(r);
+ if (reason != FSMONITOR_REASON_OK)
+ return reason;
+ }
+
+ return FSMONITOR_REASON_OK;
+}
--
gitgitgadget
^ permalink raw reply related [flat|nested] 89+ messages in thread* [PATCH v5 2/6] fsmonitor: determine if filesystem is local or remote
2022-12-12 21:57 ` [PATCH v5 0/6] fsmonitor: Implement fsmonitor " Eric DeCosta via GitGitGadget
2022-12-12 21:58 ` [PATCH v5 1/6] fsmonitor: prepare to share code between Mac OS and Linux Eric DeCosta via GitGitGadget
@ 2022-12-12 21:58 ` Eric DeCosta via GitGitGadget
2022-12-12 21:58 ` [PATCH v5 3/6] fsmonitor: implement filesystem change listener for Linux Eric DeCosta via GitGitGadget
` (4 subsequent siblings)
6 siblings, 0 replies; 89+ messages in thread
From: Eric DeCosta via GitGitGadget @ 2022-12-12 21:58 UTC (permalink / raw)
To: git
Cc: Eric Sunshine, Ævar Arnfjörð Bjarmason, Glen Choo,
Johannes Schindelin, Taylor Blau, Eric DeCosta, Eric DeCosta
From: Eric DeCosta <edecosta@mathworks.com>
Compare the given path to the mounted filesystems. Find the mount that is
the longest prefix of the path (if any) and determine if that mount is on a
local or remote filesystem.
Signed-off-by: Eric DeCosta <edecosta@mathworks.com>
---
compat/fsmonitor/fsm-path-utils-linux.c | 193 ++++++++++++++++++++++++
1 file changed, 193 insertions(+)
create mode 100644 compat/fsmonitor/fsm-path-utils-linux.c
diff --git a/compat/fsmonitor/fsm-path-utils-linux.c b/compat/fsmonitor/fsm-path-utils-linux.c
new file mode 100644
index 00000000000..629731f75f6
--- /dev/null
+++ b/compat/fsmonitor/fsm-path-utils-linux.c
@@ -0,0 +1,193 @@
+#include "fsmonitor.h"
+#include "fsmonitor-path-utils.h"
+#include <errno.h>
+#include <mntent.h>
+#include <sys/mount.h>
+#include <sys/vfs.h>
+#include <sys/statvfs.h>
+
+static int is_remote_fs(const char *path) {
+ struct statfs fs;
+
+ if (statfs(path, &fs)) {
+ error_errno(_("statfs('%s') failed"), path);
+ return -1;
+ }
+
+ switch (fs.f_type) {
+ case 0x61636673: /* ACFS */
+ case 0x5346414F: /* AFS */
+ case 0x00C36400: /* CEPH */
+ case 0xFF534D42: /* CIFS */
+ case 0x73757245: /* CODA */
+ case 0x19830326: /* FHGFS */
+ case 0x1161970: /* GFS */
+ case 0x47504653: /* GPFS */
+ case 0x013111A8: /* IBRIX */
+ case 0x6B414653: /* KAFS */
+ case 0x0BD00BD0: /* LUSTRE */
+ case 0x564C: /* NCP */
+ case 0x6969: /* NFS */
+ case 0x6E667364: /* NFSD */
+ case 0x7461636f: /* OCFS2 */
+ case 0xAAD7AAEA: /* PANFS */
+ case 0x517B: /* SMB */
+ case 0xBEEFDEAD: /* SNFS */
+ case 0xFE534D42: /* SMB2 */
+ case 0xBACBACBC: /* VMHGFS */
+ case 0xA501FCF5: /* VXFS */
+ return 1;
+ default:
+ break;
+ }
+
+ return 0;
+}
+
+static int find_mount(const char *path, const struct statvfs *fs,
+ struct mntent *entry)
+{
+ const char *const mounts = "/proc/mounts";
+ char *rp = real_pathdup(path, 1);
+ struct mntent *ment = NULL;
+ struct statvfs mntfs;
+ FILE *fp;
+ int found = 0;
+ int dlen, plen, flen = 0;
+
+ entry->mnt_fsname = NULL;
+ entry->mnt_dir = NULL;
+ entry->mnt_type = NULL;
+
+ fp = setmntent(mounts, "r");
+ if (!fp) {
+ free(rp);
+ error_errno(_("setmntent('%s') failed"), mounts);
+ return -1;
+ }
+
+ plen = strlen(rp);
+
+ /* read all the mount information and compare to path */
+ while ((ment = getmntent(fp)) != NULL) {
+ if (statvfs(ment->mnt_dir, &mntfs)) {
+ switch (errno) {
+ case EPERM:
+ case ESRCH:
+ case EACCES:
+ continue;
+ default:
+ error_errno(_("statvfs('%s') failed"), ment->mnt_dir);
+ endmntent(fp);
+ free(rp);
+ return -1;
+ }
+ }
+
+ /* is mount on the same filesystem and is a prefix of the path */
+ if ((fs->f_fsid == mntfs.f_fsid) &&
+ !strncmp(ment->mnt_dir, rp, strlen(ment->mnt_dir))) {
+ dlen = strlen(ment->mnt_dir);
+ if (dlen > plen)
+ continue;
+ /*
+ * look for the longest prefix (including root)
+ */
+ if (dlen > flen &&
+ ((dlen == 1 && ment->mnt_dir[0] == '/') ||
+ (!rp[dlen] || rp[dlen] == '/'))) {
+ flen = dlen;
+ found = 1;
+
+ /*
+ * https://man7.org/linux/man-pages/man3/getmntent.3.html
+ *
+ * The pointer points to a static area of memory which is
+ * overwritten by subsequent calls to getmntent().
+ */
+ free(entry->mnt_fsname);
+ free(entry->mnt_dir);
+ free(entry->mnt_type);
+ entry->mnt_fsname = xstrdup(ment->mnt_fsname);
+ entry->mnt_dir = xstrdup(ment->mnt_dir);
+ entry->mnt_type = xstrdup(ment->mnt_type);
+ }
+ }
+ }
+ endmntent(fp);
+ free(rp);
+
+ if (!found)
+ return -1;
+
+ return 0;
+}
+
+int fsmonitor__get_fs_info(const char *path, struct fs_info *fs_info)
+{
+ struct mntent entry;
+ struct statvfs fs;
+
+ fs_info->is_remote = -1;
+ fs_info->typename = NULL;
+
+ if (statvfs(path, &fs))
+ return error_errno(_("statvfs('%s') failed"), path);
+
+ if (find_mount(path, &fs, &entry) < 0) {
+ free(entry.mnt_fsname);
+ free(entry.mnt_dir);
+ free(entry.mnt_type);
+ return -1;
+ }
+
+ trace_printf_key(&trace_fsmonitor,
+ "statvfs('%s') [flags 0x%08lx] '%s' '%s'",
+ path, fs.f_flag, entry.mnt_type, entry.mnt_fsname);
+
+ fs_info->is_remote = is_remote_fs(entry.mnt_dir);
+ fs_info->typename = xstrdup(entry.mnt_fsname);
+ free(entry.mnt_fsname);
+ free(entry.mnt_dir);
+ free(entry.mnt_type);
+
+ if (fs_info->is_remote < 0)
+ return -1;
+
+ trace_printf_key(&trace_fsmonitor,
+ "'%s' is_remote: %d",
+ path, fs_info->is_remote);
+
+ return 0;
+}
+
+int fsmonitor__is_fs_remote(const char *path)
+{
+ struct fs_info fs;
+
+ if (fsmonitor__get_fs_info(path, &fs)) {
+ free(fs.typename);
+ return -1;
+ }
+
+ free(fs.typename);
+
+ return fs.is_remote;
+}
+
+/*
+ * No-op for now.
+ */
+int fsmonitor__get_alias(const char *path, struct alias_info *info)
+{
+ return 0;
+}
+
+/*
+ * No-op for now.
+ */
+char *fsmonitor__resolve_alias(const char *path,
+ const struct alias_info *info)
+{
+ return NULL;
+}
--
gitgitgadget
^ permalink raw reply related [flat|nested] 89+ messages in thread* [PATCH v5 3/6] fsmonitor: implement filesystem change listener for Linux
2022-12-12 21:57 ` [PATCH v5 0/6] fsmonitor: Implement fsmonitor " Eric DeCosta via GitGitGadget
2022-12-12 21:58 ` [PATCH v5 1/6] fsmonitor: prepare to share code between Mac OS and Linux Eric DeCosta via GitGitGadget
2022-12-12 21:58 ` [PATCH v5 2/6] fsmonitor: determine if filesystem is local or remote Eric DeCosta via GitGitGadget
@ 2022-12-12 21:58 ` Eric DeCosta via GitGitGadget
2022-12-12 21:58 ` [PATCH v5 4/6] fsmonitor: enable fsmonitor " Eric DeCosta via GitGitGadget
` (3 subsequent siblings)
6 siblings, 0 replies; 89+ messages in thread
From: Eric DeCosta via GitGitGadget @ 2022-12-12 21:58 UTC (permalink / raw)
To: git
Cc: Eric Sunshine, Ævar Arnfjörð Bjarmason, Glen Choo,
Johannes Schindelin, Taylor Blau, Eric DeCosta, Eric DeCosta
From: Eric DeCosta <edecosta@mathworks.com>
Implement a filesystem change listener for Linux based on the inotify API:
https://man7.org/linux/man-pages/man7/inotify.7.html
inotify requires registering a watch on every directory in the worktree and
special handling of moves/renames.
Signed-off-by: Eric DeCosta <edecosta@mathworks.com>
---
compat/fsmonitor/fsm-listen-linux.c | 676 ++++++++++++++++++++++++++++
1 file changed, 676 insertions(+)
create mode 100644 compat/fsmonitor/fsm-listen-linux.c
diff --git a/compat/fsmonitor/fsm-listen-linux.c b/compat/fsmonitor/fsm-listen-linux.c
new file mode 100644
index 00000000000..e8548e4e009
--- /dev/null
+++ b/compat/fsmonitor/fsm-listen-linux.c
@@ -0,0 +1,676 @@
+#include "cache.h"
+#include "fsmonitor.h"
+#include "fsm-listen.h"
+#include "fsmonitor--daemon.h"
+#include <dirent.h>
+#include <fcntl.h>
+#include <sys/inotify.h>
+#include <sys/stat.h>
+
+/*
+ * Safe value to bitwise OR with rest of mask for
+ * kernels that do not support IN_MASK_CREATE
+ */
+#ifndef IN_MASK_CREATE
+#define IN_MASK_CREATE 0x00000000
+#endif
+
+enum shutdown_reason {
+ SHUTDOWN_CONTINUE = 0,
+ SHUTDOWN_STOP,
+ SHUTDOWN_ERROR,
+ SHUTDOWN_FORCE
+};
+
+struct watch_entry {
+ struct hashmap_entry ent;
+ int wd;
+ uint32_t cookie;
+ const char *dir;
+};
+
+struct rename_entry {
+ struct hashmap_entry ent;
+ time_t whence;
+ uint32_t cookie;
+ const char *dir;
+};
+
+struct fsm_listen_data {
+ int fd_inotify;
+ enum shutdown_reason shutdown;
+ struct hashmap watches;
+ struct hashmap renames;
+ struct hashmap revwatches;
+};
+
+static int watch_entry_cmp(const void *cmp_data,
+ const struct hashmap_entry *eptr,
+ const struct hashmap_entry *entry_or_key,
+ const void *keydata)
+{
+ const struct watch_entry *e1, *e2;
+
+ e1 = container_of(eptr, const struct watch_entry, ent);
+ e2 = container_of(eptr, const struct watch_entry, ent);
+ return e1->wd != e2->wd;
+}
+
+static int revwatches_entry_cmp(const void *cmp_data,
+ const struct hashmap_entry *eptr,
+ const struct hashmap_entry *entry_or_key,
+ const void *keydata)
+{
+ const struct watch_entry *e1, *e2;
+
+ e1 = container_of(eptr, const struct watch_entry, ent);
+ e2 = container_of(eptr, const struct watch_entry, ent);
+ return strcmp(e1->dir, e2->dir);
+}
+
+static int rename_entry_cmp(const void *cmp_data,
+ const struct hashmap_entry *eptr,
+ const struct hashmap_entry *entry_or_key,
+ const void *keydata)
+{
+ const struct rename_entry *e1, *e2;
+
+ e1 = container_of(eptr, const struct rename_entry, ent);
+ e2 = container_of(eptr, const struct rename_entry, ent);
+ return e1->cookie != e2->cookie;
+}
+
+/*
+ * Register an inotify watch, add watch descriptor to path mapping
+ * and the reverse mapping.
+ */
+static int add_watch(const char *path, struct fsm_listen_data *data)
+{
+ const char *interned = strintern(path);
+ struct watch_entry *w1, *w2;
+
+ /* add the inotify watch, don't allow watches to be modified */
+ int wd = inotify_add_watch(data->fd_inotify, interned,
+ (IN_ALL_EVENTS | IN_ONLYDIR | IN_MASK_CREATE)
+ ^ IN_ACCESS ^ IN_CLOSE ^ IN_OPEN);
+ if (wd < 0)
+ return error_errno("inotify_add_watch('%s') failed", interned);
+
+ /* add watch descriptor -> directory mapping */
+ CALLOC_ARRAY(w1, 1);
+ w1->wd = wd;
+ w1->dir = interned;
+ hashmap_entry_init(&w1->ent, memhash(&w1->wd, sizeof(int)));
+ hashmap_add(&data->watches, &w1->ent);
+
+ /* add directory -> watch descriptor mapping */
+ CALLOC_ARRAY(w2, 1);
+ w2->wd = wd;
+ w2->dir = interned;
+ hashmap_entry_init(&w2->ent, memhash(w2->dir, strlen(w2->dir)));
+ hashmap_add(&data->revwatches, &w2->ent);
+
+ return 0;
+}
+
+/*
+ * Remove the inotify watch, the watch descriptor to path mapping
+ * and the reverse mapping.
+ */
+static void remove_watch(struct watch_entry *w,
+ struct fsm_listen_data *data)
+{
+ struct watch_entry k1, k2, *w1, *w2;
+
+ /* remove watch, ignore error if kernel already did it */
+ if (inotify_rm_watch(data->fd_inotify, w->wd) && errno != EINVAL)
+ error_errno("inotify_rm_watch() failed");
+
+ hashmap_entry_init(&k1.ent, memhash(&w->wd, sizeof(int)));
+ w1 = hashmap_remove_entry(&data->watches, &k1, ent, NULL);
+ if (!w1)
+ BUG("Double remove of watch for '%s'", w->dir);
+
+ if (w1->cookie)
+ BUG("Removing watch for '%s' which has a pending rename", w1->dir);
+
+ hashmap_entry_init(&k2.ent, memhash(w->dir, strlen(w->dir)));
+ w2 = hashmap_remove_entry(&data->revwatches, &k2, ent, NULL);
+ if (!w2)
+ BUG("Double remove of reverse watch for '%s'", w->dir);
+
+ /* w1->dir and w2->dir are interned strings, we don't own them */
+ free(w1);
+ free(w2);
+}
+
+/*
+ * Check for stale directory renames.
+ *
+ * https://man7.org/linux/man-pages/man7/inotify.7.html
+ *
+ * Allow for some small timeout to account for the fact that insertion of the
+ * IN_MOVED_FROM+IN_MOVED_TO event pair is not atomic, and the possibility that
+ * there may not be any IN_MOVED_TO event.
+ *
+ * If the IN_MOVED_TO event is not received within the timeout then events have
+ * been missed and the monitor is in an inconsistent state with respect to the
+ * filesystem.
+ */
+static int check_stale_dir_renames(struct hashmap *renames, time_t max_age)
+{
+ struct rename_entry *re;
+ struct hashmap_iter iter;
+
+ hashmap_for_each_entry(renames, &iter, re, ent) {
+ if (re->whence <= max_age)
+ return -1;
+ }
+ return 0;
+}
+
+/*
+ * Track pending renames.
+ *
+ * Tracking is done via a event cookie to watch descriptor mapping.
+ *
+ * A rename is not complete until matching a IN_MOVED_TO event is received
+ * for a corresponding IN_MOVED_FROM event.
+ */
+static void add_dir_rename(uint32_t cookie, const char *path,
+ struct fsm_listen_data *data)
+{
+ struct watch_entry k, *w;
+ struct rename_entry *re;
+
+ /* lookup the watch descriptor for the given path */
+ hashmap_entry_init(&k.ent, memhash(path, strlen(path)));
+ w = hashmap_get_entry(&data->revwatches, &k, ent, NULL);
+ if (!w) /* should never happen */
+ BUG("No watch for '%s'", path);
+ w->cookie = cookie;
+
+ /* add the pending rename to match against later */
+ CALLOC_ARRAY(re, 1);
+ re->dir = w->dir;
+ re->cookie = w->cookie;
+ re->whence = time(NULL);
+ hashmap_entry_init(&re->ent, memhash(&re->cookie, sizeof(uint32_t)));
+ hashmap_add(&data->renames, &re->ent);
+}
+
+/*
+ * Handle directory renames
+ *
+ * Once a IN_MOVED_TO event is received, lookup the rename tracking information
+ * via the event cookie and use this information to update the watch.
+ */
+static void rename_dir(uint32_t cookie, const char *path,
+ struct fsm_listen_data *data)
+{
+ struct rename_entry rek, *re;
+ struct watch_entry k, *w;
+
+ /* lookup a pending rename to match */
+ rek.cookie = cookie;
+ hashmap_entry_init(&rek.ent, memhash(&rek.cookie, sizeof(uint32_t)));
+ re = hashmap_get_entry(&data->renames, &rek, ent, NULL);
+ if (re) {
+ k.dir = re->dir;
+ hashmap_entry_init(&k.ent, memhash(k.dir, strlen(k.dir)));
+ w = hashmap_get_entry(&data->revwatches, &k, ent, NULL);
+ if (w) {
+ w->cookie = 0; /* rename handled */
+ remove_watch(w, data);
+ add_watch(path, data);
+ } else {
+ BUG("No matching watch");
+ }
+ } else {
+ BUG("No matching cookie");
+ }
+}
+
+/*
+ * Recursively add watches to every directory under path
+ */
+static int register_inotify(const char *path,
+ struct fsmonitor_daemon_state *state,
+ struct fsmonitor_batch *batch)
+{
+ DIR *dir;
+ const char *rel;
+ struct strbuf current = STRBUF_INIT;
+ struct dirent *de;
+ struct stat fs;
+ int ret = -1;
+
+ dir = opendir(path);
+ if (!dir)
+ return error_errno("opendir('%s') failed", path);
+
+ while ((de = readdir_skip_dot_and_dotdot(dir)) != NULL) {
+ strbuf_reset(¤t);
+ strbuf_addf(¤t, "%s/%s", path, de->d_name);
+ if (lstat(current.buf, &fs)) {
+ error_errno("lstat('%s') failed", current.buf);
+ goto failed;
+ }
+
+ /* recurse into directory */
+ if (S_ISDIR(fs.st_mode)) {
+ if (add_watch(current.buf, state->listen_data))
+ goto failed;
+ if (register_inotify(current.buf, state, batch))
+ goto failed;
+ } else if (batch) {
+ rel = current.buf + state->path_worktree_watch.len + 1;
+ trace_printf_key(&trace_fsmonitor, "explicitly adding '%s'", rel);
+ fsmonitor_batch__add_path(batch, rel);
+ }
+ }
+ ret = 0;
+
+failed:
+ strbuf_release(¤t);
+ if (closedir(dir) < 0)
+ return error_errno("closedir('%s') failed", path);
+ return ret;
+}
+
+static int em_rename_dir_from(u_int32_t mask)
+{
+ return ((mask & IN_ISDIR) && (mask & IN_MOVED_FROM));
+}
+
+static int em_rename_dir_to(u_int32_t mask)
+{
+ return ((mask & IN_ISDIR) && (mask & IN_MOVED_TO));
+}
+
+static int em_remove_watch(u_int32_t mask)
+{
+ return (mask & IN_DELETE_SELF);
+}
+
+static int em_dir_renamed(u_int32_t mask)
+{
+ return ((mask & IN_ISDIR) && (mask & IN_MOVE));
+}
+
+static int em_dir_created(u_int32_t mask)
+{
+ return ((mask & IN_ISDIR) && (mask & IN_CREATE));
+}
+
+static int em_dir_deleted(uint32_t mask)
+{
+ return ((mask & IN_ISDIR) && (mask & IN_DELETE));
+}
+
+static int em_force_shutdown(u_int32_t mask)
+{
+ return (mask & IN_UNMOUNT) || (mask & IN_Q_OVERFLOW);
+}
+
+static int em_ignore(u_int32_t mask)
+{
+ return (mask & IN_IGNORED) || (mask & IN_MOVE_SELF);
+}
+
+static void log_mask_set(const char *path, u_int32_t mask)
+{
+ struct strbuf msg = STRBUF_INIT;
+
+ if (mask & IN_ACCESS)
+ strbuf_addstr(&msg, "IN_ACCESS|");
+ if (mask & IN_MODIFY)
+ strbuf_addstr(&msg, "IN_MODIFY|");
+ if (mask & IN_ATTRIB)
+ strbuf_addstr(&msg, "IN_ATTRIB|");
+ if (mask & IN_CLOSE_WRITE)
+ strbuf_addstr(&msg, "IN_CLOSE_WRITE|");
+ if (mask & IN_CLOSE_NOWRITE)
+ strbuf_addstr(&msg, "IN_CLOSE_NOWRITE|");
+ if (mask & IN_OPEN)
+ strbuf_addstr(&msg, "IN_OPEN|");
+ if (mask & IN_MOVED_FROM)
+ strbuf_addstr(&msg, "IN_MOVED_FROM|");
+ if (mask & IN_MOVED_TO)
+ strbuf_addstr(&msg, "IN_MOVED_TO|");
+ if (mask & IN_CREATE)
+ strbuf_addstr(&msg, "IN_CREATE|");
+ if (mask & IN_DELETE)
+ strbuf_addstr(&msg, "IN_DELETE|");
+ if (mask & IN_DELETE_SELF)
+ strbuf_addstr(&msg, "IN_DELETE_SELF|");
+ if (mask & IN_MOVE_SELF)
+ strbuf_addstr(&msg, "IN_MOVE_SELF|");
+ if (mask & IN_UNMOUNT)
+ strbuf_addstr(&msg, "IN_UNMOUNT|");
+ if (mask & IN_Q_OVERFLOW)
+ strbuf_addstr(&msg, "IN_Q_OVERFLOW|");
+ if (mask & IN_IGNORED)
+ strbuf_addstr(&msg, "IN_IGNORED|");
+ if (mask & IN_ISDIR)
+ strbuf_addstr(&msg, "IN_ISDIR|");
+
+ trace_printf_key(&trace_fsmonitor, "inotify_event: '%s', mask=%#8.8x %s",
+ path, mask, msg.buf);
+
+ strbuf_release(&msg);
+}
+
+int fsm_listen__ctor(struct fsmonitor_daemon_state *state)
+{
+ int fd;
+ int ret = 0;
+ struct fsm_listen_data *data;
+
+ CALLOC_ARRAY(data, 1);
+ state->listen_data = data;
+ state->listen_error_code = -1;
+ data->shutdown = SHUTDOWN_ERROR;
+
+ fd = inotify_init1(O_NONBLOCK);
+ if (fd < 0)
+ return error_errno("inotify_init1() failed");
+
+ data->fd_inotify = fd;
+
+ hashmap_init(&data->watches, watch_entry_cmp, NULL, 0);
+ hashmap_init(&data->renames, rename_entry_cmp, NULL, 0);
+ hashmap_init(&data->revwatches, revwatches_entry_cmp, NULL, 0);
+
+ if (add_watch(state->path_worktree_watch.buf, data))
+ ret = -1;
+ else if (register_inotify(state->path_worktree_watch.buf, state, NULL))
+ ret = -1;
+ else if (state->nr_paths_watching > 1) {
+ if (add_watch(state->path_gitdir_watch.buf, data))
+ ret = -1;
+ else if (register_inotify(state->path_gitdir_watch.buf, state, NULL))
+ ret = -1;
+ }
+
+ if (!ret) {
+ state->listen_error_code = 0;
+ data->shutdown = SHUTDOWN_CONTINUE;
+ }
+
+ return ret;
+}
+
+void fsm_listen__dtor(struct fsmonitor_daemon_state *state)
+{
+ struct fsm_listen_data *data;
+ struct hashmap_iter iter;
+ struct watch_entry *w;
+ int fd;
+
+ if (!state || !state->listen_data)
+ return;
+
+ data = state->listen_data;
+ fd = data->fd_inotify;
+
+ hashmap_for_each_entry(&data->watches, &iter, w, ent) {
+ w->cookie = 0; /* ignore any pending renames */
+ remove_watch(w, data);
+ }
+ hashmap_clear(&data->watches);
+
+ hashmap_clear(&data->revwatches); /* remove_watch freed the entries */
+
+ hashmap_clear_and_free(&data->renames, struct rename_entry, ent);
+
+ FREE_AND_NULL(state->listen_data);
+
+ if (fd && (close(fd) < 0))
+ error_errno(_("closing inotify file descriptor failed"));
+}
+
+void fsm_listen__stop_async(struct fsmonitor_daemon_state *state)
+{
+ if (!state->listen_data->shutdown)
+ state->listen_data->shutdown = SHUTDOWN_STOP;
+}
+
+/*
+ * Process a single inotify event and queue for publication.
+ */
+static int process_event(const char *path,
+ const struct inotify_event *event,
+ struct fsmonitor_batch *batch,
+ struct string_list *cookie_list,
+ struct fsmonitor_daemon_state *state)
+{
+ const char *rel;
+ const char *last_sep;
+
+ switch (fsmonitor_classify_path_absolute(state, path)) {
+ case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX:
+ case IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX:
+ /* Use just the filename of the cookie file. */
+ last_sep = find_last_dir_sep(path);
+ string_list_append(cookie_list,
+ last_sep ? last_sep + 1 : path);
+ break;
+ case IS_INSIDE_DOT_GIT:
+ case IS_INSIDE_GITDIR:
+ break;
+ case IS_DOT_GIT:
+ case IS_GITDIR:
+ /*
+ * If .git directory is deleted or renamed away,
+ * we have to quit.
+ */
+ if (em_dir_deleted(event->mask)) {
+ trace_printf_key(&trace_fsmonitor,
+ "event: gitdir removed");
+ state->listen_data->shutdown = SHUTDOWN_FORCE;
+ goto done;
+ }
+
+ if (em_dir_renamed(event->mask)) {
+ trace_printf_key(&trace_fsmonitor,
+ "event: gitdir renamed");
+ state->listen_data->shutdown = SHUTDOWN_FORCE;
+ goto done;
+ }
+ break;
+ case IS_WORKDIR_PATH:
+ /* normal events in the working directory */
+ if (trace_pass_fl(&trace_fsmonitor))
+ log_mask_set(path, event->mask);
+
+ rel = path + state->path_worktree_watch.len + 1;
+ fsmonitor_batch__add_path(batch, rel);
+
+ if (em_dir_deleted(event->mask))
+ break;
+
+ /* received IN_MOVE_FROM, add tracking for expected IN_MOVE_TO */
+ if (em_rename_dir_from(event->mask))
+ add_dir_rename(event->cookie, path, state->listen_data);
+
+ /* received IN_MOVE_TO, update watch to reflect new path */
+ if (em_rename_dir_to(event->mask)) {
+ rename_dir(event->cookie, path, state->listen_data);
+ if (register_inotify(path, state, batch)) {
+ state->listen_data->shutdown = SHUTDOWN_ERROR;
+ goto done;
+ }
+ }
+
+ if (em_dir_created(event->mask)) {
+ if (add_watch(path, state->listen_data)) {
+ state->listen_data->shutdown = SHUTDOWN_ERROR;
+ goto done;
+ }
+ if (register_inotify(path, state, batch)) {
+ state->listen_data->shutdown = SHUTDOWN_ERROR;
+ goto done;
+ }
+ }
+ break;
+ case IS_OUTSIDE_CONE:
+ default:
+ trace_printf_key(&trace_fsmonitor,
+ "ignoring '%s'", path);
+ break;
+ }
+ return 0;
+done:
+ return -1;
+}
+
+/*
+ * Read the inotify event stream and pre-process events before further
+ * processing and eventual publishing.
+ */
+static void handle_events(struct fsmonitor_daemon_state *state)
+{
+ /* See https://man7.org/linux/man-pages/man7/inotify.7.html */
+ char buf[4096]
+ __attribute__ ((aligned(__alignof__(struct inotify_event))));
+
+ struct hashmap watches = state->listen_data->watches;
+ struct fsmonitor_batch *batch = NULL;
+ struct string_list cookie_list = STRING_LIST_INIT_DUP;
+ struct watch_entry k, *w;
+ struct strbuf path;
+ const struct inotify_event *event;
+ int fd = state->listen_data->fd_inotify;
+ ssize_t len;
+ char *ptr, *p;
+
+ strbuf_init(&path, PATH_MAX);
+
+ for(;;) {
+ len = read(fd, buf, sizeof(buf));
+ if (len == -1 && errno != EAGAIN) {
+ error_errno(_("reading inotify message stream failed"));
+ state->listen_data->shutdown = SHUTDOWN_ERROR;
+ goto done;
+ }
+
+ /* nothing to read */
+ if (len <= 0)
+ goto done;
+
+ /* Loop over all events in the buffer. */
+ for (ptr = buf; ptr < buf + len;
+ ptr += sizeof(struct inotify_event) + event->len) {
+
+ event = (const struct inotify_event *) ptr;
+
+ if (em_ignore(event->mask))
+ continue;
+
+ /* File system was unmounted or event queue overflowed */
+ if (em_force_shutdown(event->mask)) {
+ if (trace_pass_fl(&trace_fsmonitor))
+ log_mask_set("Forcing shutdown", event->mask);
+ state->listen_data->shutdown = SHUTDOWN_FORCE;
+ goto done;
+ }
+
+ hashmap_entry_init(&k.ent, memhash(&event->wd, sizeof(int)));
+ k.wd = event->wd;
+
+ w = hashmap_get_entry(&watches, &k, ent, NULL);
+ if (!w) /* should never happen */
+ BUG("No watch for '%s'", event->name);
+
+ /* directory watch was removed */
+ if (em_remove_watch(event->mask)) {
+ remove_watch(w, state->listen_data);
+ continue;
+ }
+
+ strbuf_reset(&path);
+ strbuf_add(&path, w->dir, strlen(w->dir));
+ strbuf_addch(&path, '/');
+ strbuf_addstr(&path, event->name);
+
+ p = fsmonitor__resolve_alias(path.buf, &state->alias);
+ if (!p)
+ p = strbuf_detach(&path, NULL);
+
+ if (!batch)
+ batch = fsmonitor_batch__new();
+
+ if (process_event(p, event, batch, &cookie_list, state)) {
+ free(p);
+ goto done;
+ }
+ free(p);
+ }
+ strbuf_reset(&path);
+ fsmonitor_publish(state, batch, &cookie_list);
+ string_list_clear(&cookie_list, 0);
+ batch = NULL;
+ }
+done:
+ strbuf_release(&path);
+ fsmonitor_batch__free_list(batch);
+ string_list_clear(&cookie_list, 0);
+}
+
+/*
+ * Non-blocking read of the inotify events stream. The inotify fd is polled
+ * frequently to help minimize the number of queue overflows.
+ */
+void fsm_listen__loop(struct fsmonitor_daemon_state *state)
+{
+ int poll_num;
+ const int interval = 1000;
+ time_t checked = time(NULL);
+ struct pollfd fds[1];
+ fds[0].fd = state->listen_data->fd_inotify;
+ fds[0].events = POLLIN;
+
+ for(;;) {
+ switch (state->listen_data->shutdown) {
+ case SHUTDOWN_CONTINUE:
+ poll_num = poll(fds, 1, 1);
+ if (poll_num == -1) {
+ if (errno == EINTR)
+ continue;
+ error_errno(_("polling inotify message stream failed"));
+ state->listen_data->shutdown = SHUTDOWN_ERROR;
+ continue;
+ }
+
+ if ((time(NULL) - checked) >= interval) {
+ checked = time(NULL);
+ if (check_stale_dir_renames(&state->listen_data->renames,
+ checked - interval)) {
+ trace_printf_key(&trace_fsmonitor,
+ "Missed IN_MOVED_TO events, forcing shutdown");
+ state->listen_data->shutdown = SHUTDOWN_FORCE;
+ continue;
+ }
+ }
+
+ if (poll_num > 0 && (fds[0].revents & POLLIN))
+ handle_events(state);
+
+ continue;
+ case SHUTDOWN_ERROR:
+ state->listen_error_code = -1;
+ ipc_server_stop_async(state->ipc_server_data);
+ break;
+ case SHUTDOWN_FORCE:
+ state->listen_error_code = 0;
+ ipc_server_stop_async(state->ipc_server_data);
+ break;
+ case SHUTDOWN_STOP:
+ default:
+ state->listen_error_code = 0;
+ break;
+ }
+ return;
+ }
+}
--
gitgitgadget
^ permalink raw reply related [flat|nested] 89+ messages in thread* [PATCH v5 4/6] fsmonitor: enable fsmonitor for Linux
2022-12-12 21:57 ` [PATCH v5 0/6] fsmonitor: Implement fsmonitor " Eric DeCosta via GitGitGadget
` (2 preceding siblings ...)
2022-12-12 21:58 ` [PATCH v5 3/6] fsmonitor: implement filesystem change listener for Linux Eric DeCosta via GitGitGadget
@ 2022-12-12 21:58 ` Eric DeCosta via GitGitGadget
2022-12-12 21:58 ` [PATCH v5 5/6] fsmonitor: test updates Eric DeCosta via GitGitGadget
` (2 subsequent siblings)
6 siblings, 0 replies; 89+ messages in thread
From: Eric DeCosta via GitGitGadget @ 2022-12-12 21:58 UTC (permalink / raw)
To: git
Cc: Eric Sunshine, Ævar Arnfjörð Bjarmason, Glen Choo,
Johannes Schindelin, Taylor Blau, Eric DeCosta, Eric DeCosta
From: Eric DeCosta <edecosta@mathworks.com>
Update build to enable fsmonitor for Linux.
Signed-off-by: Eric DeCosta <edecosta@mathworks.com>
---
config.mak.uname | 8 ++++++++
contrib/buildsystems/CMakeLists.txt | 11 ++++++++++-
2 files changed, 18 insertions(+), 1 deletion(-)
diff --git a/config.mak.uname b/config.mak.uname
index d63629fe807..5890d810463 100644
--- a/config.mak.uname
+++ b/config.mak.uname
@@ -68,6 +68,14 @@ ifeq ($(uname_S),Linux)
ifneq ($(findstring .el7.,$(uname_R)),)
BASIC_CFLAGS += -std=c99
endif
+ # The builtin FSMonitor on Linux builds upon Simple-IPC. Both require
+ # Unix domain sockets and PThreads.
+ ifndef NO_PTHREADS
+ ifndef NO_UNIX_SOCKETS
+ FSMONITOR_DAEMON_BACKEND = linux
+ FSMONITOR_OS_SETTINGS = linux
+ endif
+ endif
endif
ifeq ($(uname_S),GNU/kFreeBSD)
HAVE_ALLOCA_H = YesPlease
diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt
index 2f6e0197ffa..a0319527279 100644
--- a/contrib/buildsystems/CMakeLists.txt
+++ b/contrib/buildsystems/CMakeLists.txt
@@ -304,7 +304,16 @@ else()
endif()
if(SUPPORTS_SIMPLE_IPC)
- if(CMAKE_SYSTEM_NAME STREQUAL "Windows")
+ if(CMAKE_SYSTEM_NAME STREQUAL "Linux")
+ add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)
+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-linux.c)
+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-linux.c)
+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-ipc-linux.c)
+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-path-utils-linux.c)
+
+ add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS)
+ list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-linux.c)
+ elseif(CMAKE_SYSTEM_NAME STREQUAL "Windows")
add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND)
list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c)
list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-win32.c)
--
gitgitgadget
^ permalink raw reply related [flat|nested] 89+ messages in thread* [PATCH v5 5/6] fsmonitor: test updates
2022-12-12 21:57 ` [PATCH v5 0/6] fsmonitor: Implement fsmonitor " Eric DeCosta via GitGitGadget
` (3 preceding siblings ...)
2022-12-12 21:58 ` [PATCH v5 4/6] fsmonitor: enable fsmonitor " Eric DeCosta via GitGitGadget
@ 2022-12-12 21:58 ` Eric DeCosta via GitGitGadget
2022-12-12 21:58 ` [PATCH v5 6/6] fsmonitor: update doc for Linux Eric DeCosta via GitGitGadget
2023-04-12 5:42 ` [PATCH v5 0/6] fsmonitor: Implement fsmonitor " Junio C Hamano
6 siblings, 0 replies; 89+ messages in thread
From: Eric DeCosta via GitGitGadget @ 2022-12-12 21:58 UTC (permalink / raw)
To: git
Cc: Eric Sunshine, Ævar Arnfjörð Bjarmason, Glen Choo,
Johannes Schindelin, Taylor Blau, Eric DeCosta, Eric DeCosta
From: Eric DeCosta <edecosta@mathworks.com>
t7527-builtin-fsmonitor was leaking fsmonitor--daemon processes in some
cases.
Accomodate slight difference in the number of events generated on Linux.
On lower-powered systems, spin a little to give the daemon time
to respond to and log filesystem events.
Signed-off-by: Eric DeCosta <edecosta@mathworks.com>
---
t/t7527-builtin-fsmonitor.sh | 94 ++++++++++++++++++++++++++++++------
1 file changed, 80 insertions(+), 14 deletions(-)
diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh
index 4abc74db2bb..951374231b7 100755
--- a/t/t7527-builtin-fsmonitor.sh
+++ b/t/t7527-builtin-fsmonitor.sh
@@ -13,7 +13,7 @@ fi
stop_daemon_delete_repo () {
r=$1 &&
test_might_fail git -C $r fsmonitor--daemon stop &&
- rm -rf $1
+ rm -rf $r
}
start_daemon () {
@@ -72,6 +72,34 @@ start_daemon () {
)
}
+IMPLICIT_TIMEOUT=5
+
+wait_for_update () {
+ func=$1 &&
+ file=$2 &&
+ sz=$(wc -c < "$file") &&
+ last=0 &&
+ $func &&
+ k=0 &&
+ while test "$k" -lt $IMPLICIT_TIMEOUT
+ do
+ nsz=$(wc -c < "$file")
+ if test "$nsz" -gt "$sz"
+ then
+ if test "$last" -eq "$nsz"
+ then
+ cat "$file" &&
+ return 0
+ fi
+ last=$nsz
+ fi
+ sleep 1
+ k=$(( $k + 1 ))
+ done &&
+ cat "$file" &&
+ return 0
+}
+
# Is a Trace2 data event present with the given catetory and key?
# We do not care what the value is.
#
@@ -137,7 +165,6 @@ test_expect_success 'implicit daemon start' '
# machines (where it might take a moment to wake and reschedule the
# daemon process) to avoid false alarms during test runs.)
#
-IMPLICIT_TIMEOUT=5
verify_implicit_shutdown () {
r=$1 &&
@@ -373,6 +400,15 @@ create_files () {
echo 3 >dir2/new
}
+rename_directory () {
+ mv dirtorename dirrenamed
+}
+
+rename_directory_file () {
+ mv dirtorename dirrenamed &&
+ echo 1 > dirrenamed/new
+}
+
rename_files () {
mv rename renamed &&
mv dir1/rename dir1/renamed &&
@@ -427,10 +463,12 @@ test_expect_success 'edit some files' '
start_daemon --tf "$PWD/.git/trace" &&
- edit_files &&
+ wait_for_update edit_files "$PWD/.git/trace" &&
test-tool fsmonitor-client query --token 0 &&
+ test_might_fail git fsmonitor--daemon stop &&
+
grep "^event: dir1/modified$" .git/trace &&
grep "^event: dir2/modified$" .git/trace &&
grep "^event: modified$" .git/trace &&
@@ -442,10 +480,12 @@ test_expect_success 'create some files' '
start_daemon --tf "$PWD/.git/trace" &&
- create_files &&
+ wait_for_update create_files "$PWD/.git/trace" &&
test-tool fsmonitor-client query --token 0 &&
+ test_might_fail git fsmonitor--daemon stop &&
+
grep "^event: dir1/new$" .git/trace &&
grep "^event: dir2/new$" .git/trace &&
grep "^event: new$" .git/trace
@@ -456,10 +496,12 @@ test_expect_success 'delete some files' '
start_daemon --tf "$PWD/.git/trace" &&
- delete_files &&
+ wait_for_update delete_files "$PWD/.git/trace" &&
test-tool fsmonitor-client query --token 0 &&
+ test_might_fail git fsmonitor--daemon stop &&
+
grep "^event: dir1/delete$" .git/trace &&
grep "^event: dir2/delete$" .git/trace &&
grep "^event: delete$" .git/trace
@@ -470,10 +512,12 @@ test_expect_success 'rename some files' '
start_daemon --tf "$PWD/.git/trace" &&
- rename_files &&
+ wait_for_update rename_files "$PWD/.git/trace" &&
test-tool fsmonitor-client query --token 0 &&
+ test_might_fail git fsmonitor--daemon stop &&
+
grep "^event: dir1/rename$" .git/trace &&
grep "^event: dir2/rename$" .git/trace &&
grep "^event: rename$" .git/trace &&
@@ -487,23 +531,42 @@ test_expect_success 'rename directory' '
start_daemon --tf "$PWD/.git/trace" &&
- mv dirtorename dirrenamed &&
+ wait_for_update rename_directory "$PWD/.git/trace" &&
test-tool fsmonitor-client query --token 0 &&
+ test_might_fail git fsmonitor--daemon stop &&
+
grep "^event: dirtorename/*$" .git/trace &&
grep "^event: dirrenamed/*$" .git/trace
'
+test_expect_success 'rename directory file' '
+ test_when_finished clean_up_repo_and_stop_daemon &&
+
+ start_daemon --tf "$PWD/.git/trace" &&
+
+ wait_for_update rename_directory_file "$PWD/.git/trace" &&
+
+ test-tool fsmonitor-client query --token 0 &&
+
+ test_might_fail git fsmonitor--daemon stop &&
+
+ grep "^event: dirtorename/*$" .git/trace &&
+ grep "^event: dirrenamed/*$" .git/trace &&
+ grep "^event: dirrenamed/new$" .git/trace
+'
test_expect_success 'file changes to directory' '
test_when_finished clean_up_repo_and_stop_daemon &&
start_daemon --tf "$PWD/.git/trace" &&
- file_to_directory &&
+ wait_for_update file_to_directory "$PWD/.git/trace" &&
test-tool fsmonitor-client query --token 0 &&
+ test_might_fail git fsmonitor--daemon stop &&
+
grep "^event: delete$" .git/trace &&
grep "^event: delete/new$" .git/trace
'
@@ -513,10 +576,12 @@ test_expect_success 'directory changes to a file' '
start_daemon --tf "$PWD/.git/trace" &&
- directory_to_file &&
+ wait_for_update directory_to_file "$PWD/.git/trace" &&
test-tool fsmonitor-client query --token 0 &&
+ test_might_fail git fsmonitor--daemon stop &&
+
grep "^event: dir1$" .git/trace
'
@@ -561,7 +626,7 @@ test_expect_success 'flush cached data' '
test-tool -C test_flush fsmonitor-client query --token "builtin:test_00000002:0" >actual_2 &&
nul_to_q <actual_2 >actual_q2 &&
- grep "^builtin:test_00000002:0Q$" actual_q2 &&
+ grep "^builtin:test_00000002:[0-1]Q$" actual_q2 &&
>test_flush/file_3 &&
@@ -732,7 +797,8 @@ u_values="$u1 $u2"
for u in $u_values
do
test_expect_success "unicode in repo root path: $u" '
- test_when_finished "stop_daemon_delete_repo $u" &&
+ test_when_finished \
+ "stop_daemon_delete_repo `echo "$u" | sed 's:x:\\\\\\\\\\\\\\x:g'`" &&
git init "$u" &&
echo 1 >"$u"/file1 &&
@@ -818,8 +884,7 @@ test_expect_success 'submodule setup' '
'
test_expect_success 'submodule always visited' '
- test_when_finished "git -C super fsmonitor--daemon stop; \
- rm -rf super; \
+ test_when_finished "rm -rf super; \
rm -rf sub" &&
create_super super &&
@@ -887,7 +952,8 @@ have_t2_error_event () {
}
test_expect_success "stray submodule super-prefix warning" '
- test_when_finished "rm -rf super; \
+ test_when_finished "git -C super/dir_1/dir_2/sub fsmonitor--daemon stop; \
+ rm -rf super; \
rm -rf sub; \
rm super-sub.trace" &&
--
gitgitgadget
^ permalink raw reply related [flat|nested] 89+ messages in thread* [PATCH v5 6/6] fsmonitor: update doc for Linux
2022-12-12 21:57 ` [PATCH v5 0/6] fsmonitor: Implement fsmonitor " Eric DeCosta via GitGitGadget
` (4 preceding siblings ...)
2022-12-12 21:58 ` [PATCH v5 5/6] fsmonitor: test updates Eric DeCosta via GitGitGadget
@ 2022-12-12 21:58 ` Eric DeCosta via GitGitGadget
2023-04-12 5:42 ` [PATCH v5 0/6] fsmonitor: Implement fsmonitor " Junio C Hamano
6 siblings, 0 replies; 89+ messages in thread
From: Eric DeCosta via GitGitGadget @ 2022-12-12 21:58 UTC (permalink / raw)
To: git
Cc: Eric Sunshine, Ævar Arnfjörð Bjarmason, Glen Choo,
Johannes Schindelin, Taylor Blau, Eric DeCosta, Eric DeCosta
From: Eric DeCosta <edecosta@mathworks.com>
Update the documentation for Linux.
Signed-off-by: Eric DeCosta <edecosta@mathworks.com>
---
Documentation/config/fsmonitor--daemon.txt | 4 ++--
Documentation/git-fsmonitor--daemon.txt | 24 ++++++++++++++--------
2 files changed, 18 insertions(+), 10 deletions(-)
diff --git a/Documentation/config/fsmonitor--daemon.txt b/Documentation/config/fsmonitor--daemon.txt
index c225c6c9e74..2cafb040d96 100644
--- a/Documentation/config/fsmonitor--daemon.txt
+++ b/Documentation/config/fsmonitor--daemon.txt
@@ -4,8 +4,8 @@ fsmonitor.allowRemote::
behavior. Only respected when `core.fsmonitor` is set to `true`.
fsmonitor.socketDir::
- This Mac OS-specific option, if set, specifies the directory in
+ Mac OS and Linux-specific option. If set, specifies the directory in
which to create the Unix domain socket used for communication
between the fsmonitor daemon and various Git commands. The directory must
- reside on a native Mac OS filesystem. Only respected when `core.fsmonitor`
+ reside on a native filesystem. Only respected when `core.fsmonitor`
is set to `true`.
diff --git a/Documentation/git-fsmonitor--daemon.txt b/Documentation/git-fsmonitor--daemon.txt
index 8238eadb0e1..c2b08229c74 100644
--- a/Documentation/git-fsmonitor--daemon.txt
+++ b/Documentation/git-fsmonitor--daemon.txt
@@ -76,23 +76,31 @@ repositories; this may be overridden by setting `fsmonitor.allowRemote` to
correctly with all network-mounted repositories and such use is considered
experimental.
-On Mac OS, the inter-process communication (IPC) between various Git
+On Linux and Mac OS, the inter-process communication (IPC) between various Git
commands and the fsmonitor daemon is done via a Unix domain socket (UDS) -- a
-special type of file -- which is supported by native Mac OS filesystems,
-but not on network-mounted filesystems, NTFS, or FAT32. Other filesystems
-may or may not have the needed support; the fsmonitor daemon is not guaranteed
-to work with these filesystems and such use is considered experimental.
+special type of file -- which is supported by many native Linux and Mac OS
+filesystems, but not on network-mounted filesystems, NTFS, or FAT32. Other
+filesystems may or may not have the needed support; the fsmonitor daemon is not
+guaranteed to work with these filesystems and such use is considered
+experimental.
By default, the socket is created in the `.git` directory, however, if the
`.git` directory is on a network-mounted filesystem, it will be instead be
created at `$HOME/.git-fsmonitor-*` unless `$HOME` itself is on a
network-mounted filesystem in which case you must set the configuration
-variable `fsmonitor.socketDir` to the path of a directory on a Mac OS native
+variable `fsmonitor.socketDir` to the path of a directory on a native
filesystem in which to create the socket file.
If none of the above directories (`.git`, `$HOME`, or `fsmonitor.socketDir`)
-is on a native Mac OS file filesystem the fsmonitor daemon will report an
-error that will cause the daemon and the currently running command to exit.
+is on a native Linux or Mac OS filesystem the fsmonitor daemon will report
+an error that will cause the daemon to exit and the currently running command
+to issue a warning.
+
+On Linux, the fsmonitor daemon registers a watch for each directory in the
+repository. The default per-user limit for the number of watches on most Linux
+systems is 8192. This may not be sufficient for large repositories or if
+multiple instances of the fsmonitor daemon are running.
+See https://watchexec.github.io/docs/inotify-limits.html[Linux inotify limits] for more information.
CONFIGURATION
-------------
--
gitgitgadget
^ permalink raw reply related [flat|nested] 89+ messages in thread* Re: [PATCH v5 0/6] fsmonitor: Implement fsmonitor for Linux
2022-12-12 21:57 ` [PATCH v5 0/6] fsmonitor: Implement fsmonitor " Eric DeCosta via GitGitGadget
` (5 preceding siblings ...)
2022-12-12 21:58 ` [PATCH v5 6/6] fsmonitor: update doc for Linux Eric DeCosta via GitGitGadget
@ 2023-04-12 5:42 ` Junio C Hamano
6 siblings, 0 replies; 89+ messages in thread
From: Junio C Hamano @ 2023-04-12 5:42 UTC (permalink / raw)
To: Eric DeCosta via GitGitGadget
Cc: git, Eric Sunshine, Ævar Arnfjörð Bjarmason,
Glen Choo, Johannes Schindelin, Taylor Blau, Eric DeCosta
"Eric DeCosta via GitGitGadget" <gitgitgadget@gmail.com> writes:
> Goal is to deliver fsmonitor for Linux that is on par with fsmonitor for
> Windows and Mac OS.
>
> This patch set builds upon previous work for done for Windows and Mac OS to
> implement a fsmonitor back-end for Linux based on the Linux inotify API.
> inotify differs significantly from the equivalent Windows and Mac OS APIs in
> that a watch must be registered for every directory of interest (rather than
> a singular watch at the root of the directory tree) and special care must be
> taken to handle directory renames correctly.
>
> More information about inotify:
> https://man7.org/linux/man-pages/man7/inotify.7.html
>
> v4 differs from v3:
>
> * Code review feedback
This has been dormant for full four months, and it seems to break
linux-asan CI job when merged to 'seen',
https://github.com/git/git/actions/runs/4672116751/jobs/8273938089
while the same 'seen' excluding this topic passes everything.
https://github.com/git/git/actions/runs/4674694371
For now, I'll drop this topic from 'seen'.
Thanks.
^ permalink raw reply [flat|nested] 89+ messages in thread