* [PATCH 0/2] A design for future-proofing fsync() configuration
@ 2021-12-04 3:28 Neeraj K. Singh via GitGitGadget
2021-12-04 3:28 ` [PATCH 1/2] fsync: add writeout-only mode for fsyncing repo data Neeraj Singh via GitGitGadget
` (2 more replies)
0 siblings, 3 replies; 122+ messages in thread
From: Neeraj K. Singh via GitGitGadget @ 2021-12-04 3:28 UTC (permalink / raw)
To: git; +Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps,
Neeraj K. Singh
This is an implementation of an extensible configuration mechanism for
fsyncing persistent components of a repo.
The main goals are to separate the "what" to sync from the "how". There are
now two settings: core.fsync - Control the 'what', including the index.
core.fsyncMethod - Control the 'how'. Currently we support writeout-only and
full fsync.
Syncing of refs can be layered on top of core.fsync. And batch mode will be
layered on core.fsyncMethod.
core.fsyncobjectfiles is removed and will issue a deprecation warning if
it's seen.
I'd like to get agreement on this direction before redoing batch mode on top
of this change.
As of this writing, the change isn't tested in detail. I'll be making sure
with strace that each component has the desired effect.
Please see [1], [2], and [3] for discussions that led to this series.
[1] https://lore.kernel.org/git/211110.86r1bogg27.gmgdl@evledraar.gmail.com/
[2]
https://lore.kernel.org/git/dd65718814011eb93ccc4428f9882e0f025224a6.1636029491.git.ps@pks.im/
[3]
https://lore.kernel.org/git/pull.1076.git.git.1629856292.gitgitgadget@gmail.com/
Neeraj Singh (2):
fsync: add writeout-only mode for fsyncing repo data
core.fsync: introduce granular fsync control
Documentation/config/core.txt | 35 +++++++++---
builtin/fast-import.c | 2 +-
builtin/index-pack.c | 4 +-
builtin/pack-objects.c | 8 ++-
bulk-checkin.c | 5 +-
cache.h | 48 +++++++++++++++-
commit-graph.c | 3 +-
compat/mingw.h | 3 +
compat/win32/flush.c | 28 +++++++++
config.c | 89 ++++++++++++++++++++++++++++-
config.mak.uname | 3 +
configure.ac | 8 +++
contrib/buildsystems/CMakeLists.txt | 3 +-
csum-file.c | 5 +-
csum-file.h | 2 +-
environment.c | 3 +-
git-compat-util.h | 24 ++++++++
midx.c | 3 +-
object-file.c | 3 +-
pack-bitmap-write.c | 3 +-
pack-write.c | 12 ++--
read-cache.c | 19 ++++--
wrapper.c | 56 ++++++++++++++++++
write-or-die.c | 10 ++--
24 files changed, 337 insertions(+), 42 deletions(-)
create mode 100644 compat/win32/flush.c
base-commit: abe6bb3905392d5eb6b01fa6e54d7e784e0522aa
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1093%2Fneerajsi-msft%2Fns%2Fcore-fsync-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1093/neerajsi-msft/ns/core-fsync-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/1093
--
gitgitgadget
^ permalink raw reply [flat|nested] 122+ messages in thread
* [PATCH 1/2] fsync: add writeout-only mode for fsyncing repo data
2021-12-04 3:28 [PATCH 0/2] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget
@ 2021-12-04 3:28 ` Neeraj Singh via GitGitGadget
2021-12-06 7:54 ` Neeraj Singh
2021-12-04 3:28 ` [PATCH 2/2] core.fsync: introduce granular fsync control Neeraj Singh via GitGitGadget
2021-12-07 2:46 ` [PATCH v2 0/3] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget
2 siblings, 1 reply; 122+ messages in thread
From: Neeraj Singh via GitGitGadget @ 2021-12-04 3:28 UTC (permalink / raw)
To: git
Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps,
Neeraj K. Singh, Neeraj Singh
From: Neeraj Singh <neerajsi@microsoft.com>
The new writeout-only mode attempts to tell the operating system to
flush its in-memory page cache to the storage hardware without issuing a
CACHE_FLUSH command to the storage controller.
Writeout-only fsync is significantly faster than a vanilla fsync on
common hardware, since data is written to a disk-side cache rather than
all the way to a durable medium. Later changes in this patch series will
take advantage of this primitive to implement batching of hardware
flushes.
When git_fsync is called with FSYNC_WRITEOUT_ONLY, it may fail and the
caller is expected to do an ordinary fsync as needed.
On Apple platforms, the fsync system call does not issue a CACHE_FLUSH
directive to the storage controller. This change updates fsync to do
fcntl(F_FULLFSYNC) to make fsync actually durable. We maintain parity
with existing behavior on Apple platforms by setting the default value
of the new core.fsyncmethod option.
Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
---
Documentation/config/core.txt | 9 +++++
cache.h | 7 ++++
compat/mingw.h | 3 ++
compat/win32/flush.c | 28 +++++++++++++++
config.c | 12 +++++++
config.mak.uname | 3 ++
configure.ac | 8 +++++
contrib/buildsystems/CMakeLists.txt | 3 +-
environment.c | 2 +-
git-compat-util.h | 24 +++++++++++++
wrapper.c | 56 +++++++++++++++++++++++++++++
write-or-die.c | 10 +++---
12 files changed, 159 insertions(+), 6 deletions(-)
create mode 100644 compat/win32/flush.c
diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
index c04f62a54a1..c91eccea598 100644
--- a/Documentation/config/core.txt
+++ b/Documentation/config/core.txt
@@ -547,6 +547,15 @@ core.whitespace::
is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent`
errors. The default tab width is 8. Allowed values are 1 to 63.
+core.fsyncMethod::
+ A value indicating the strategy Git will use to harden repository data
+ using fsync and related primitives.
++
+* `fsync` uses the fsync() system call or platform equivalents.
+* `writeout-only` issues requests to send the writes to the storage
+ hardware, but does not send any FLUSH CACHE request. If the operating system
+ does not support the required interfaces, this falls back to fsync().
+
core.fsyncObjectFiles::
This boolean will enable 'fsync()' when writing object files.
+
diff --git a/cache.h b/cache.h
index eba12487b99..9cd60d94952 100644
--- a/cache.h
+++ b/cache.h
@@ -986,6 +986,13 @@ extern int read_replace_refs;
extern char *git_replace_ref_base;
extern int fsync_object_files;
+
+enum fsync_method {
+ FSYNC_METHOD_FSYNC,
+ FSYNC_METHOD_WRITEOUT_ONLY
+};
+
+extern enum fsync_method fsync_method;
extern int core_preload_index;
extern int precomposed_unicode;
extern int protect_hfs;
diff --git a/compat/mingw.h b/compat/mingw.h
index c9a52ad64a6..6074a3d3ced 100644
--- a/compat/mingw.h
+++ b/compat/mingw.h
@@ -329,6 +329,9 @@ int mingw_getpagesize(void);
#define getpagesize mingw_getpagesize
#endif
+int win32_fsync_no_flush(int fd);
+#define fsync_no_flush win32_fsync_no_flush
+
struct rlimit {
unsigned int rlim_cur;
};
diff --git a/compat/win32/flush.c b/compat/win32/flush.c
new file mode 100644
index 00000000000..75324c24ee7
--- /dev/null
+++ b/compat/win32/flush.c
@@ -0,0 +1,28 @@
+#include "../../git-compat-util.h"
+#include <winternl.h>
+#include "lazyload.h"
+
+int win32_fsync_no_flush(int fd)
+{
+ IO_STATUS_BLOCK io_status;
+
+#define FLUSH_FLAGS_FILE_DATA_ONLY 1
+
+ DECLARE_PROC_ADDR(ntdll.dll, NTSTATUS, NtFlushBuffersFileEx,
+ HANDLE FileHandle, ULONG Flags, PVOID Parameters, ULONG ParameterSize,
+ PIO_STATUS_BLOCK IoStatusBlock);
+
+ if (!INIT_PROC_ADDR(NtFlushBuffersFileEx)) {
+ errno = ENOSYS;
+ return -1;
+ }
+
+ memset(&io_status, 0, sizeof(io_status));
+ if (NtFlushBuffersFileEx((HANDLE)_get_osfhandle(fd), FLUSH_FLAGS_FILE_DATA_ONLY,
+ NULL, 0, &io_status)) {
+ errno = EINVAL;
+ return -1;
+ }
+
+ return 0;
+}
diff --git a/config.c b/config.c
index c5873f3a706..c3410b8a868 100644
--- a/config.c
+++ b/config.c
@@ -1490,6 +1490,18 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
return 0;
}
+ if (!strcmp(var, "core.fsyncmethod")) {
+ if (!value)
+ return config_error_nonbool(var);
+ if (!strcmp(value, "fsync"))
+ fsync_method = FSYNC_METHOD_FSYNC;
+ else if (!strcmp(value, "writeout-only"))
+ fsync_method = FSYNC_METHOD_WRITEOUT_ONLY;
+ else
+ warning(_("unknown %s value '%s'"), var, value);
+
+ }
+
if (!strcmp(var, "core.fsyncobjectfiles")) {
fsync_object_files = git_config_bool(var, value);
return 0;
diff --git a/config.mak.uname b/config.mak.uname
index d0701f9beb0..774a09622d2 100644
--- a/config.mak.uname
+++ b/config.mak.uname
@@ -57,6 +57,7 @@ ifeq ($(uname_S),Linux)
HAVE_CLOCK_MONOTONIC = YesPlease
# -lrt is needed for clock_gettime on glibc <= 2.16
NEEDS_LIBRT = YesPlease
+ HAVE_SYNC_FILE_RANGE = YesPlease
HAVE_GETDELIM = YesPlease
FREAD_READS_DIRECTORIES = UnfortunatelyYes
BASIC_CFLAGS += -DHAVE_SYSINFO
@@ -453,6 +454,7 @@ endif
CFLAGS =
BASIC_CFLAGS = -nologo -I. -Icompat/vcbuild/include -DWIN32 -D_CONSOLE -DHAVE_STRING_H -D_CRT_SECURE_NO_WARNINGS -D_CRT_NONSTDC_NO_DEPRECATE
COMPAT_OBJS = compat/msvc.o compat/winansi.o \
+ compat/win32/flush.o \
compat/win32/path-utils.o \
compat/win32/pthread.o compat/win32/syslog.o \
compat/win32/trace2_win32_process_info.o \
@@ -628,6 +630,7 @@ ifeq ($(uname_S),MINGW)
COMPAT_CFLAGS += -DSTRIP_EXTENSION=\".exe\"
COMPAT_OBJS += compat/mingw.o compat/winansi.o \
compat/win32/trace2_win32_process_info.o \
+ compat/win32/flush.o \
compat/win32/path-utils.o \
compat/win32/pthread.o compat/win32/syslog.o \
compat/win32/dirent.o
diff --git a/configure.ac b/configure.ac
index 5ee25ec95c8..6bd6bef1c44 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1082,6 +1082,14 @@ AC_COMPILE_IFELSE([CLOCK_MONOTONIC_SRC],
[AC_MSG_RESULT([no])
HAVE_CLOCK_MONOTONIC=])
GIT_CONF_SUBST([HAVE_CLOCK_MONOTONIC])
+
+#
+# Define HAVE_SYNC_FILE_RANGE=YesPlease if sync_file_range is available.
+GIT_CHECK_FUNC(sync_file_range,
+ [HAVE_SYNC_FILE_RANGE=YesPlease],
+ [HAVE_SYNC_FILE_RANGE])
+GIT_CONF_SUBST([HAVE_SYNC_FILE_RANGE])
+
#
# Define NO_SETITIMER if you don't have setitimer.
GIT_CHECK_FUNC(setitimer,
diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt
index 86b46114464..6d7bc16d054 100644
--- a/contrib/buildsystems/CMakeLists.txt
+++ b/contrib/buildsystems/CMakeLists.txt
@@ -261,7 +261,8 @@ if(CMAKE_SYSTEM_NAME STREQUAL "Windows")
NOGDI OBJECT_CREATION_MODE=1 __USE_MINGW_ANSI_STDIO=0
USE_NED_ALLOCATOR OVERRIDE_STRDUP MMAP_PREVENTS_DELETE USE_WIN32_MMAP
UNICODE _UNICODE HAVE_WPGMPTR ENSURE_MSYSTEM_IS_SET)
- list(APPEND compat_SOURCES compat/mingw.c compat/winansi.c compat/win32/path-utils.c
+ list(APPEND compat_SOURCES compat/mingw.c compat/winansi.c
+ compat/win32/flush.c compat/win32/path-utils.c
compat/win32/pthread.c compat/win32mmap.c compat/win32/syslog.c
compat/win32/trace2_win32_process_info.c compat/win32/dirent.c
compat/nedmalloc/nedmalloc.c compat/strdup.c)
diff --git a/environment.c b/environment.c
index 9da7f3c1a19..f9140e842cf 100644
--- a/environment.c
+++ b/environment.c
@@ -41,7 +41,7 @@ const char *git_attributes_file;
const char *git_hooks_path;
int zlib_compression_level = Z_BEST_SPEED;
int pack_compression_level = Z_DEFAULT_COMPRESSION;
-int fsync_object_files;
+enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT;
size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE;
size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT;
size_t delta_base_cache_limit = 96 * 1024 * 1024;
diff --git a/git-compat-util.h b/git-compat-util.h
index c6bd2a84e55..cb9abd7a08c 100644
--- a/git-compat-util.h
+++ b/git-compat-util.h
@@ -1239,6 +1239,30 @@ __attribute__((format (printf, 1, 2))) NORETURN
void BUG(const char *fmt, ...);
#endif
+#ifdef __APPLE__
+#define FSYNC_METHOD_DEFAULT FSYNC_METHOD_WRITEOUT_ONLY
+#else
+#define FSYNC_METHOD_DEFAULT FSYNC_METHOD_FSYNC
+#endif
+
+enum fsync_action {
+ FSYNC_WRITEOUT_ONLY,
+ FSYNC_HARDWARE_FLUSH
+};
+
+/*
+ * Issues an fsync against the specified file according to the specified mode.
+ *
+ * FSYNC_WRITEOUT_ONLY attempts to use interfaces available on some operating
+ * systems to flush the OS cache without issuing a flush command to the storage
+ * controller. If those interfaces are unavailable, the function fails with
+ * ENOSYS.
+ *
+ * FSYNC_HARDWARE_FLUSH does an OS writeout and hardware flush to ensure that
+ * changes are durable. It is not expected to fail.
+ */
+int git_fsync(int fd, enum fsync_action action);
+
/*
* Preserves errno, prints a message, but gives no warning for ENOENT.
* Returns 0 on success, which includes trying to unlink an object that does
diff --git a/wrapper.c b/wrapper.c
index 36e12119d76..1c5f2c87791 100644
--- a/wrapper.c
+++ b/wrapper.c
@@ -546,6 +546,62 @@ int xmkstemp_mode(char *filename_template, int mode)
return fd;
}
+int git_fsync(int fd, enum fsync_action action)
+{
+ switch (action) {
+ case FSYNC_WRITEOUT_ONLY:
+
+#ifdef __APPLE__
+ /*
+ * on macOS, fsync just causes filesystem cache writeback but does not
+ * flush hardware caches.
+ */
+ return fsync(fd);
+#endif
+
+#ifdef HAVE_SYNC_FILE_RANGE
+ /*
+ * On linux 2.6.17 and above, sync_file_range is the way to issue
+ * a writeback without a hardware flush. An offset of 0 and size of 0
+ * indicates writeout of the entire file and the wait flags ensure that all
+ * dirty data is written to the disk (potentially in a disk-side cache)
+ * before we continue.
+ */
+
+ return sync_file_range(fd, 0, 0, SYNC_FILE_RANGE_WAIT_BEFORE |
+ SYNC_FILE_RANGE_WRITE |
+ SYNC_FILE_RANGE_WAIT_AFTER);
+#endif
+
+#ifdef fsync_no_flush
+ return fsync_no_flush(fd);
+#endif
+
+ errno = ENOSYS;
+ return -1;
+
+ case FSYNC_HARDWARE_FLUSH:
+ /*
+ * On some platforms fsync may return EINTR. Try again in this
+ * case, since callers asking for a hardware flush may die if
+ * this function returns an error.
+ */
+ for (;;) {
+ int err;
+#ifdef __APPLE__
+ err = fcntl(fd, F_FULLFSYNC);
+#else
+ err = fsync(fd);
+#endif
+ if (err >= 0 || errno != EINTR)
+ return err;
+ }
+
+ default:
+ BUG("unexpected git_fsync(%d) call", action);
+ }
+}
+
static int warn_if_unremovable(const char *op, const char *file, int rc)
{
int err;
diff --git a/write-or-die.c b/write-or-die.c
index 0b1ec8190b6..0702acdd5e8 100644
--- a/write-or-die.c
+++ b/write-or-die.c
@@ -57,10 +57,12 @@ void fprintf_or_die(FILE *f, const char *fmt, ...)
void fsync_or_die(int fd, const char *msg)
{
- while (fsync(fd) < 0) {
- if (errno != EINTR)
- die_errno("fsync error on '%s'", msg);
- }
+ if (fsync_method == FSYNC_METHOD_WRITEOUT_ONLY &&
+ git_fsync(fd, FSYNC_WRITEOUT_ONLY) >= 0)
+ return;
+
+ if (git_fsync(fd, FSYNC_HARDWARE_FLUSH) < 0)
+ die_errno("fsync error on '%s'", msg);
}
void write_or_die(int fd, const void *buf, size_t count)
--
gitgitgadget
^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH 2/2] core.fsync: introduce granular fsync control
2021-12-04 3:28 [PATCH 0/2] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget
2021-12-04 3:28 ` [PATCH 1/2] fsync: add writeout-only mode for fsyncing repo data Neeraj Singh via GitGitGadget
@ 2021-12-04 3:28 ` Neeraj Singh via GitGitGadget
2021-12-07 2:46 ` [PATCH v2 0/3] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget
2 siblings, 0 replies; 122+ messages in thread
From: Neeraj Singh via GitGitGadget @ 2021-12-04 3:28 UTC (permalink / raw)
To: git
Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps,
Neeraj K. Singh, Neeraj Singh
From: Neeraj Singh <neerajsi@microsoft.com>
This commit introduces the `core.fsync` configuration
knob which can be used to control how components of the
repository are made durable on disk.
This setting allows future extensibility of components
that could be synced in two ways:
* We issue a warning rather than an error for unrecognized
components, so new configs can be used with old Git versions.
* We support negation, so users can choose one of the default
aggregate options and then remove components that they don't
want.
This also support the common request of doing absolutely no
fysncing with the `core.fsync=none` value, which is expected
to make the test suite faster.
This commit introduces the new ability for the user to harden
the index, which is a requirement for being able to actually
find a file that has been added to the repo and then deleted
from the working tree.
Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
---
Documentation/config/core.txt | 28 +++++++++----
builtin/fast-import.c | 2 +-
builtin/index-pack.c | 4 +-
builtin/pack-objects.c | 8 ++--
bulk-checkin.c | 5 ++-
cache.h | 41 ++++++++++++++++++-
commit-graph.c | 3 +-
config.c | 77 ++++++++++++++++++++++++++++++++++-
csum-file.c | 5 ++-
csum-file.h | 2 +-
environment.c | 1 +
midx.c | 3 +-
object-file.c | 3 +-
pack-bitmap-write.c | 3 +-
pack-write.c | 12 +++---
read-cache.c | 19 ++++++---
16 files changed, 179 insertions(+), 37 deletions(-)
diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
index c91eccea598..d502f8a1bf5 100644
--- a/Documentation/config/core.txt
+++ b/Documentation/config/core.txt
@@ -547,6 +547,26 @@ core.whitespace::
is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent`
errors. The default tab width is 8. Allowed values are 1 to 63.
+core.fsync::
+ A comma-separated list of parts of the repository which should be
+ hardened via the core.fsyncMethod when created or modified. You can
+ disable hardening of any component by prefixing it with a '-'. Later
+ items take precedence over earlier ones in the list. For example,
+ `core.fsync=all,-index` means "harden everything except the index".
+ Items that are not hardened may be lost in the event of an unclean
+ system shutdown.
++
+* `none` disables fsync completely. This must be specified alone.
+* `loose-object` hardens objects added to the repo in loose-object form.
+* `pack` hardens objects added to the repo in packfile form.
+* `pack-metadata` hardens packfile bitmaps and indexes.
+* `commit-graph` hardens the commit graph file.
+* `index` hardens the index when it is modified.
+* `objects` is an aggregate option that includes `loose-objects`, `pack`,
+ `pack-metadata`, and `commit-graph`.
+* `default` is an aggregate option that is equivalent to `objects,-loose-object`
+* `all` is an aggregate option that syncs all individual components above.
+
core.fsyncMethod::
A value indicating the strategy Git will use to harden repository data
using fsync and related primitives.
@@ -556,14 +576,6 @@ core.fsyncMethod::
hardware, but does not send any FLUSH CACHE request. If the operating system
does not support the required interfaces, this falls back to fsync().
-core.fsyncObjectFiles::
- This boolean will enable 'fsync()' when writing object files.
-+
-This is a total waste of time and effort on a filesystem that orders
-data writes properly, but can be useful for filesystems that do not use
-journalling (traditional UNIX filesystems) or that only journal metadata
-and not file contents (OS X's HFS+, or Linux ext3 with "data=writeback").
-
core.preloadIndex::
Enable parallel index preload for operations like 'git diff'
+
diff --git a/builtin/fast-import.c b/builtin/fast-import.c
index 20406f67754..0ae17d63618 100644
--- a/builtin/fast-import.c
+++ b/builtin/fast-import.c
@@ -856,7 +856,7 @@ static void end_packfile(void)
struct tag *t;
close_pack_windows(pack_data);
- finalize_hashfile(pack_file, cur_pack_oid.hash, 0);
+ finalize_hashfile(pack_file, cur_pack_oid.hash, REPO_COMPONENT_PACK, 0);
fixup_pack_header_footer(pack_data->pack_fd, pack_data->hash,
pack_data->pack_name, object_count,
cur_pack_oid.hash, pack_size);
diff --git a/builtin/index-pack.c b/builtin/index-pack.c
index c23d01de7dc..9157a955de7 100644
--- a/builtin/index-pack.c
+++ b/builtin/index-pack.c
@@ -1286,7 +1286,7 @@ static void conclude_pack(int fix_thin_pack, const char *curr_pack, unsigned cha
nr_objects - nr_objects_initial);
stop_progress_msg(&progress, msg.buf);
strbuf_release(&msg);
- finalize_hashfile(f, tail_hash, 0);
+ finalize_hashfile(f, tail_hash, REPO_COMPONENT_PACK, 0);
hashcpy(read_hash, pack_hash);
fixup_pack_header_footer(output_fd, pack_hash,
curr_pack, nr_objects,
@@ -1508,7 +1508,7 @@ static void final(const char *final_pack_name, const char *curr_pack_name,
if (!from_stdin) {
close(input_fd);
} else {
- fsync_or_die(output_fd, curr_pack_name);
+ fsync_component_or_die(REPO_COMPONENT_PACK, output_fd, curr_pack_name);
err = close(output_fd);
if (err)
die_errno(_("error while closing pack file"));
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index 857be7826f3..48c2f9f3847 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -1204,11 +1204,13 @@ static void write_pack_file(void)
* If so, rewrite it like in fast-import
*/
if (pack_to_stdout) {
- finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_CLOSE);
+ finalize_hashfile(f, hash, REPO_COMPONENT_NONE,
+ CSUM_HASH_IN_STREAM | CSUM_CLOSE);
} else if (nr_written == nr_remaining) {
- finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
+ finalize_hashfile(f, hash, REPO_COMPONENT_PACK,
+ CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
} else {
- int fd = finalize_hashfile(f, hash, 0);
+ int fd = finalize_hashfile(f, hash, REPO_COMPONENT_PACK, 0);
fixup_pack_header_footer(fd, hash, pack_tmp_name,
nr_written, hash, offset);
close(fd);
diff --git a/bulk-checkin.c b/bulk-checkin.c
index 8785b2ac806..b9f3d315334 100644
--- a/bulk-checkin.c
+++ b/bulk-checkin.c
@@ -53,9 +53,10 @@ static void finish_bulk_checkin(struct bulk_checkin_state *state)
unlink(state->pack_tmp_name);
goto clear_exit;
} else if (state->nr_written == 1) {
- finalize_hashfile(state->f, hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
+ finalize_hashfile(state->f, hash, REPO_COMPONENT_PACK,
+ CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
} else {
- int fd = finalize_hashfile(state->f, hash, 0);
+ int fd = finalize_hashfile(state->f, hash, REPO_COMPONENT_PACK, 0);
fixup_pack_header_footer(fd, hash, state->pack_tmp_name,
state->nr_written, hash,
state->offset);
diff --git a/cache.h b/cache.h
index 9cd60d94952..b2966352440 100644
--- a/cache.h
+++ b/cache.h
@@ -985,7 +985,40 @@ void reset_shared_repository(void);
extern int read_replace_refs;
extern char *git_replace_ref_base;
-extern int fsync_object_files;
+/*
+ * These values are used to help identify parts of a repository to fsync.
+ * REPO_COMPONENT_NONE identifies data that will not be a persistent part of the
+ * repository and so shouldn't be fsynced.
+ */
+enum repo_component {
+ REPO_COMPONENT_NONE = 0,
+ REPO_COMPONENT_LOOSE_OBJECT = 1 << 0,
+ REPO_COMPONENT_PACK = 1 << 1,
+ REPO_COMPONENT_PACK_METADATA = 1 << 2,
+ REPO_COMPONENT_COMMIT_GRAPH = 1 << 3,
+ REPO_COMPONENT_INDEX = 1 << 4,
+};
+
+#define FSYNC_COMPONENTS_DEFAULT (REPO_COMPONENT_PACK | \
+ REPO_COMPONENT_PACK_METADATA | \
+ REPO_COMPONENT_COMMIT_GRAPH)
+
+#define FSYNC_COMPONENTS_OBJECTS (REPO_COMPONENT_LOOSE_OBJECT | \
+ REPO_COMPONENT_PACK | \
+ REPO_COMPONENT_PACK_METADATA | \
+ REPO_COMPONENT_COMMIT_GRAPH)
+
+#define FSYNC_COMPONENTS_ALL (REPO_COMPONENT_LOOSE_OBJECT | \
+ REPO_COMPONENT_PACK | \
+ REPO_COMPONENT_PACK_METADATA | \
+ REPO_COMPONENT_COMMIT_GRAPH | \
+ REPO_COMPONENT_INDEX)
+
+
+/*
+ * A bitmask indicating which components of the repo should be fsynced.
+ */
+extern enum repo_component fsync_components;
enum fsync_method {
FSYNC_METHOD_FSYNC,
@@ -1747,6 +1780,12 @@ int copy_file_with_time(const char *dst, const char *src, int mode);
void write_or_die(int fd, const void *buf, size_t count);
void fsync_or_die(int fd, const char *);
+inline void fsync_component_or_die(enum repo_component component, int fd, const char *msg)
+{
+ if (fsync_components & component)
+ fsync_or_die(fd, msg);
+}
+
ssize_t read_in_full(int fd, void *buf, size_t count);
ssize_t write_in_full(int fd, const void *buf, size_t count);
ssize_t pread_in_full(int fd, void *buf, size_t count, off_t offset);
diff --git a/commit-graph.c b/commit-graph.c
index 2706683acfe..4bed4175ab2 100644
--- a/commit-graph.c
+++ b/commit-graph.c
@@ -1939,7 +1939,8 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)
}
close_commit_graph(ctx->r->objects);
- finalize_hashfile(f, file_hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC);
+ finalize_hashfile(f, file_hash, REPO_COMPONENT_COMMIT_GRAPH,
+ CSUM_HASH_IN_STREAM | CSUM_FSYNC);
free_chunkfile(cf);
if (ctx->split) {
diff --git a/config.c b/config.c
index c3410b8a868..6c8b102ed7a 100644
--- a/config.c
+++ b/config.c
@@ -1213,6 +1213,74 @@ static int git_parse_maybe_bool_text(const char *value)
return -1;
}
+static const struct fsync_component_entry {
+ const char *name;
+ enum repo_component component_bits;
+} fsync_component_table[] = {
+ { "loose-object", REPO_COMPONENT_LOOSE_OBJECT },
+ { "pack", REPO_COMPONENT_PACK },
+ { "pack-metadata", REPO_COMPONENT_PACK_METADATA },
+ { "commit-graph", REPO_COMPONENT_COMMIT_GRAPH },
+ { "index", REPO_COMPONENT_INDEX },
+ { "objects", FSYNC_COMPONENTS_OBJECTS },
+ { "default", FSYNC_COMPONENTS_DEFAULT },
+ { "all", FSYNC_COMPONENTS_ALL },
+};
+
+static enum repo_component parse_fsync_components(const char *var, const char *string)
+{
+ enum repo_component output = 0;
+
+ if (!strcmp(string, "none"))
+ return output;
+
+ while (string) {
+ int i;
+ size_t len;
+ const char *ep;
+ int negated = 0;
+ int found = 0;
+
+ string = string + strspn(string, ", \t\n\r");
+ ep = strchrnul(string, ',');
+ len = ep - string;
+
+ if (*string == '-') {
+ negated = 1;
+ string++;
+ len--;
+ if (!len)
+ warning(_("invalid value for variable %s"), var);
+ }
+
+ if (!len)
+ break;
+
+ for (i = 0; i < ARRAY_SIZE(fsync_component_table); ++i) {
+ const struct fsync_component_entry *entry = &fsync_component_table[i];
+
+ if (strncmp(entry->name, string, len))
+ continue;
+
+ found = 1;
+ if (negated)
+ output &= ~entry->component_bits;
+ else
+ output |= entry->component_bits;
+ }
+
+ if (!found) {
+ char *component = xstrndup(string, len);
+ warning(_("unknown %s value '%s'"), var, component);
+ free(component);
+ }
+
+ string = ep;
+ }
+
+ return output;
+}
+
int git_parse_maybe_bool(const char *value)
{
int v = git_parse_maybe_bool_text(value);
@@ -1490,6 +1558,13 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
return 0;
}
+ if (!strcmp(var, "core.fsync")) {
+ if (!value)
+ return config_error_nonbool(var);
+ fsync_components = parse_fsync_components(var, value);
+ return 0;
+ }
+
if (!strcmp(var, "core.fsyncmethod")) {
if (!value)
return config_error_nonbool(var);
@@ -1503,7 +1578,7 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
}
if (!strcmp(var, "core.fsyncobjectfiles")) {
- fsync_object_files = git_config_bool(var, value);
+ warning(_("core.fsyncobjectfiles is deprecated; use core.fsync instead"));
return 0;
}
diff --git a/csum-file.c b/csum-file.c
index 26e8a6df44e..adc8023d5af 100644
--- a/csum-file.c
+++ b/csum-file.c
@@ -58,7 +58,8 @@ static void free_hashfile(struct hashfile *f)
free(f);
}
-int finalize_hashfile(struct hashfile *f, unsigned char *result, unsigned int flags)
+int finalize_hashfile(struct hashfile *f, unsigned char *result,
+ enum repo_component component, unsigned int flags)
{
int fd;
@@ -69,7 +70,7 @@ int finalize_hashfile(struct hashfile *f, unsigned char *result, unsigned int fl
if (flags & CSUM_HASH_IN_STREAM)
flush(f, f->buffer, the_hash_algo->rawsz);
if (flags & CSUM_FSYNC)
- fsync_or_die(f->fd, f->name);
+ fsync_component_or_die(component, f->fd, f->name);
if (flags & CSUM_CLOSE) {
if (close(f->fd))
die_errno("%s: sha1 file error on close", f->name);
diff --git a/csum-file.h b/csum-file.h
index 291215b34eb..3820b4a0e94 100644
--- a/csum-file.h
+++ b/csum-file.h
@@ -38,7 +38,7 @@ int hashfile_truncate(struct hashfile *, struct hashfile_checkpoint *);
struct hashfile *hashfd(int fd, const char *name);
struct hashfile *hashfd_check(const char *name);
struct hashfile *hashfd_throughput(int fd, const char *name, struct progress *tp);
-int finalize_hashfile(struct hashfile *, unsigned char *, unsigned int);
+int finalize_hashfile(struct hashfile *, unsigned char *, enum repo_component, unsigned int);
void hashwrite(struct hashfile *, const void *, unsigned int);
void hashflush(struct hashfile *f);
void crc32_begin(struct hashfile *);
diff --git a/environment.c b/environment.c
index f9140e842cf..190df463475 100644
--- a/environment.c
+++ b/environment.c
@@ -42,6 +42,7 @@ const char *git_hooks_path;
int zlib_compression_level = Z_BEST_SPEED;
int pack_compression_level = Z_DEFAULT_COMPRESSION;
enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT;
+enum repo_component fsync_components = FSYNC_COMPONENTS_DEFAULT;
size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE;
size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT;
size_t delta_base_cache_limit = 96 * 1024 * 1024;
diff --git a/midx.c b/midx.c
index 837b46b2af5..6e9510ab0dc 100644
--- a/midx.c
+++ b/midx.c
@@ -1406,7 +1406,8 @@ static int write_midx_internal(const char *object_dir,
write_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs);
write_chunkfile(cf, &ctx);
- finalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);
+ finalize_hashfile(f, midx_hash, REPO_COMPONENT_PACK_METADATA,
+ CSUM_FSYNC | CSUM_HASH_IN_STREAM);
free_chunkfile(cf);
if (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP))
diff --git a/object-file.c b/object-file.c
index eb972cdccd2..871ea6bcd53 100644
--- a/object-file.c
+++ b/object-file.c
@@ -1809,8 +1809,7 @@ int hash_object_file(const struct git_hash_algo *algo, const void *buf,
/* Finalize a file on disk, and close it. */
static void close_loose_object(int fd)
{
- if (fsync_object_files)
- fsync_or_die(fd, "loose object file");
+ fsync_component_or_die(REPO_COMPONENT_LOOSE_OBJECT, fd, "loose object file");
if (close(fd) != 0)
die_errno(_("error when closing loose object file"));
}
diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c
index 9c55c1531e1..e82e87af996 100644
--- a/pack-bitmap-write.c
+++ b/pack-bitmap-write.c
@@ -719,7 +719,8 @@ void bitmap_writer_finish(struct pack_idx_entry **index,
if (options & BITMAP_OPT_HASH_CACHE)
write_hash_cache(f, index, index_nr);
- finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
+ finalize_hashfile(f, NULL, REPO_COMPONENT_PACK_METADATA,
+ CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
if (adjust_shared_perm(tmp_file.buf))
die_errno("unable to make temporary bitmap file readable");
diff --git a/pack-write.c b/pack-write.c
index a5846f3a346..d9c37803e98 100644
--- a/pack-write.c
+++ b/pack-write.c
@@ -159,8 +159,9 @@ const char *write_idx_file(const char *index_name, struct pack_idx_entry **objec
}
hashwrite(f, sha1, the_hash_algo->rawsz);
- finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE |
- ((opts->flags & WRITE_IDX_VERIFY)
+ finalize_hashfile(f, NULL, REPO_COMPONENT_PACK_METADATA,
+ CSUM_HASH_IN_STREAM | CSUM_CLOSE |
+ ((opts->flags & WRITE_IDX_VERIFY)
? 0 : CSUM_FSYNC));
return index_name;
}
@@ -281,8 +282,9 @@ const char *write_rev_file_order(const char *rev_name,
if (rev_name && adjust_shared_perm(rev_name) < 0)
die(_("failed to make %s readable"), rev_name);
- finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE |
- ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC));
+ finalize_hashfile(f, NULL, REPO_COMPONENT_PACK_METADATA,
+ CSUM_HASH_IN_STREAM | CSUM_CLOSE |
+ ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC));
return rev_name;
}
@@ -390,7 +392,7 @@ void fixup_pack_header_footer(int pack_fd,
the_hash_algo->final_fn(partial_pack_hash, &old_hash_ctx);
the_hash_algo->final_fn(new_pack_hash, &new_hash_ctx);
write_or_die(pack_fd, new_pack_hash, the_hash_algo->rawsz);
- fsync_or_die(pack_fd, pack_name);
+ fsync_component_or_die(REPO_COMPONENT_PACK, pack_fd, pack_name);
}
char *index_pack_lockfile(int ip_out, int *is_well_formed)
diff --git a/read-cache.c b/read-cache.c
index f3986596623..883d0c0019a 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -2816,7 +2816,7 @@ static int record_ieot(void)
* rely on it.
*/
static int do_write_index(struct index_state *istate, struct tempfile *tempfile,
- int strip_extensions)
+ int strip_extensions, unsigned flags)
{
uint64_t start = getnanotime();
struct hashfile *f;
@@ -2830,6 +2830,7 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile,
struct strbuf previous_name_buf = STRBUF_INIT, *previous_name;
int drop_cache_tree = istate->drop_cache_tree;
off_t offset;
+ int csum_fsync_flag;
int ieot_entries = 1;
struct index_entry_offset_table *ieot = NULL;
int nr, nr_threads;
@@ -3060,7 +3061,13 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile,
return -1;
}
- finalize_hashfile(f, istate->oid.hash, CSUM_HASH_IN_STREAM);
+ csum_fsync_flag = 0;
+ if (!alternate_index_output && (flags & COMMIT_LOCK))
+ csum_fsync_flag = CSUM_FSYNC;
+
+ finalize_hashfile(f, istate->oid.hash, REPO_COMPONENT_INDEX,
+ CSUM_HASH_IN_STREAM | csum_fsync_flag);
+
if (close_tempfile_gently(tempfile)) {
error(_("could not close '%s'"), get_tempfile_path(tempfile));
return -1;
@@ -3115,7 +3122,7 @@ static int do_write_locked_index(struct index_state *istate, struct lock_file *l
*/
trace2_region_enter_printf("index", "do_write_index", the_repository,
"%s", get_lock_file_path(lock));
- ret = do_write_index(istate, lock->tempfile, 0);
+ ret = do_write_index(istate, lock->tempfile, 0, flags);
trace2_region_leave_printf("index", "do_write_index", the_repository,
"%s", get_lock_file_path(lock));
@@ -3209,7 +3216,7 @@ static int clean_shared_index_files(const char *current_hex)
}
static int write_shared_index(struct index_state *istate,
- struct tempfile **temp)
+ struct tempfile **temp, unsigned flags)
{
struct split_index *si = istate->split_index;
int ret, was_full = !istate->sparse_index;
@@ -3219,7 +3226,7 @@ static int write_shared_index(struct index_state *istate,
trace2_region_enter_printf("index", "shared/do_write_index",
the_repository, "%s", get_tempfile_path(*temp));
- ret = do_write_index(si->base, *temp, 1);
+ ret = do_write_index(si->base, *temp, 1, flags);
trace2_region_leave_printf("index", "shared/do_write_index",
the_repository, "%s", get_tempfile_path(*temp));
@@ -3328,7 +3335,7 @@ int write_locked_index(struct index_state *istate, struct lock_file *lock,
ret = do_write_locked_index(istate, lock, flags);
goto out;
}
- ret = write_shared_index(istate, &temp);
+ ret = write_shared_index(istate, &temp, flags);
saved_errno = errno;
if (is_tempfile_active(temp))
--
gitgitgadget
^ permalink raw reply related [flat|nested] 122+ messages in thread
* Re: [PATCH 1/2] fsync: add writeout-only mode for fsyncing repo data
2021-12-04 3:28 ` [PATCH 1/2] fsync: add writeout-only mode for fsyncing repo data Neeraj Singh via GitGitGadget
@ 2021-12-06 7:54 ` Neeraj Singh
0 siblings, 0 replies; 122+ messages in thread
From: Neeraj Singh @ 2021-12-06 7:54 UTC (permalink / raw)
To: Neeraj Singh via GitGitGadget
Cc: git, rsbecker, bagasdotme, newren, avarab, ps, Neeraj K. Singh
Context for reviewers: The mechanism part of this patch is equivalent to [1]
and [2]. With the following changes:
* The configuration code is a bit different and we now expose a
`writeout-only` mode to the user. This mode is the default on macOS
to prevent a change in end-user behavior.
* git_fsync now contains the EINTR retry loop internally.
[1] https://lore.kernel.org/git/e1747ce00af7ab3170a69955b07d995d5321d6f3.1637020263.git.gitgitgadget@gmail.com/
[2] https://lore.kernel.org/git/546ad9c82e8e0c2eb4683f9f360d8f30e2136020.1630108177.git.gitgitgadget@gmail.com/
^ permalink raw reply [flat|nested] 122+ messages in thread
* [PATCH v2 0/3] A design for future-proofing fsync() configuration
2021-12-04 3:28 [PATCH 0/2] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget
2021-12-04 3:28 ` [PATCH 1/2] fsync: add writeout-only mode for fsyncing repo data Neeraj Singh via GitGitGadget
2021-12-04 3:28 ` [PATCH 2/2] core.fsync: introduce granular fsync control Neeraj Singh via GitGitGadget
@ 2021-12-07 2:46 ` Neeraj K. Singh via GitGitGadget
2021-12-07 2:46 ` [PATCH v2 1/3] core.fsyncmethod: add writeout-only mode Neeraj Singh via GitGitGadget
` (4 more replies)
2 siblings, 5 replies; 122+ messages in thread
From: Neeraj K. Singh via GitGitGadget @ 2021-12-07 2:46 UTC (permalink / raw)
To: git; +Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps,
Neeraj K. Singh
This is an implementation of an extensible configuration mechanism for
fsyncing persistent components of a repo.
The main goals are to separate the "what" to sync from the "how". There are
now two settings: core.fsync - Control the 'what', including the index.
core.fsyncMethod - Control the 'how'. Currently we support writeout-only and
full fsync.
Syncing of refs can be layered on top of core.fsync. And batch mode will be
layered on core.fsyncMethod.
core.fsyncobjectfiles is removed and will issue a deprecation warning if
it's seen.
I'd like to get agreement on this direction before submitting batch mode to
the list. The batch mode series is available to view at
https://github.com/neerajsi-msft/git/pull/1.
Please see [1], [2], and [3] for discussions that led to this series.
V2 changes:
* Updated the documentation for core.fsyncmethod to be less certain.
writeout-only probably does not do the right thing on Linux.
* Split out the core.fsync=index change into its own commit.
* Rename REPO_COMPONENT to FSYNC_COMPONENT. This is really specific to
fsyncing, so the name should reflect that.
* Re-add missing Makefile change for SYNC_FILE_RANGE.
* Tested writeout-only mode, index syncing, and general config settings.
[1] https://lore.kernel.org/git/211110.86r1bogg27.gmgdl@evledraar.gmail.com/
[2]
https://lore.kernel.org/git/dd65718814011eb93ccc4428f9882e0f025224a6.1636029491.git.ps@pks.im/
[3]
https://lore.kernel.org/git/pull.1076.git.git.1629856292.gitgitgadget@gmail.com/
Neeraj Singh (3):
core.fsyncmethod: add writeout-only mode
core.fsync: introduce granular fsync control
core.fsync: new option to harden the index
Documentation/config/core.txt | 35 +++++++++---
Makefile | 6 ++
builtin/fast-import.c | 2 +-
builtin/index-pack.c | 4 +-
builtin/pack-objects.c | 8 ++-
bulk-checkin.c | 5 +-
cache.h | 48 +++++++++++++++-
commit-graph.c | 3 +-
compat/mingw.h | 3 +
compat/win32/flush.c | 28 +++++++++
config.c | 89 ++++++++++++++++++++++++++++-
config.mak.uname | 3 +
configure.ac | 8 +++
contrib/buildsystems/CMakeLists.txt | 3 +-
csum-file.c | 5 +-
csum-file.h | 3 +-
environment.c | 3 +-
git-compat-util.h | 24 ++++++++
midx.c | 3 +-
object-file.c | 3 +-
pack-bitmap-write.c | 3 +-
pack-write.c | 13 +++--
read-cache.c | 19 ++++--
wrapper.c | 56 ++++++++++++++++++
write-or-die.c | 10 ++--
25 files changed, 344 insertions(+), 43 deletions(-)
create mode 100644 compat/win32/flush.c
base-commit: abe6bb3905392d5eb6b01fa6e54d7e784e0522aa
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1093%2Fneerajsi-msft%2Fns%2Fcore-fsync-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1093/neerajsi-msft/ns/core-fsync-v2
Pull-Request: https://github.com/gitgitgadget/git/pull/1093
Range-diff vs v1:
1: 527380ddc3f ! 1: e79522cbdd4 fsync: add writeout-only mode for fsyncing repo data
@@ Metadata
Author: Neeraj Singh <neerajsi@microsoft.com>
## Commit message ##
- fsync: add writeout-only mode for fsyncing repo data
+ core.fsyncmethod: add writeout-only mode
+
+ This commit introduces the `core.fsyncmethod` configuration
+ knob, which can currently be set to `fsync` or `writeout-only`.
The new writeout-only mode attempts to tell the operating system to
flush its in-memory page cache to the storage hardware without issuing a
@@ Documentation/config/core.txt: core.whitespace::
+ using fsync and related primitives.
++
+* `fsync` uses the fsync() system call or platform equivalents.
-+* `writeout-only` issues requests to send the writes to the storage
-+ hardware, but does not send any FLUSH CACHE request. If the operating system
-+ does not support the required interfaces, this falls back to fsync().
++* `writeout-only` issues pagecache writeback requests, but depending on the
++ filesystem and storage hardware, data added to the repository may not be
++ durable in the event of a system crash. This is the default mode on macOS.
+
core.fsyncObjectFiles::
This boolean will enable 'fsync()' when writing object files.
+
+ ## Makefile ##
+@@ Makefile: all::
+ #
+ # Define HAVE_CLOCK_MONOTONIC if your platform has CLOCK_MONOTONIC.
+ #
++# Define HAVE_SYNC_FILE_RANGE if your platform has sync_file_range.
++#
+ # Define NEEDS_LIBRT if your platform requires linking with librt (glibc version
+ # before 2.17) for clock_gettime and CLOCK_MONOTONIC.
+ #
+@@ Makefile: ifdef HAVE_CLOCK_MONOTONIC
+ BASIC_CFLAGS += -DHAVE_CLOCK_MONOTONIC
+ endif
+
++ifdef HAVE_SYNC_FILE_RANGE
++ BASIC_CFLAGS += -DHAVE_SYNC_FILE_RANGE
++endif
++
+ ifdef NEEDS_LIBRT
+ EXTLIBS += -lrt
+ endif
+
## cache.h ##
@@ cache.h: extern int read_replace_refs;
extern char *git_replace_ref_base;
2: 23311a10142 ! 2: ff80a94bf9a core.fsync: introduce granular fsync control
@@ Commit message
knob which can be used to control how components of the
repository are made durable on disk.
- This setting allows future extensibility of components
- that could be synced in two ways:
+ This setting allows future extensibility of the list of
+ syncable components:
* We issue a warning rather than an error for unrecognized
components, so new configs can be used with old Git versions.
* We support negation, so users can choose one of the default
aggregate options and then remove components that they don't
- want.
+ want. The user would then harden any new components added in
+ a Git version update.
- This also support the common request of doing absolutely no
+ This also supports the common request of doing absolutely no
fysncing with the `core.fsync=none` value, which is expected
to make the test suite faster.
- This commit introduces the new ability for the user to harden
- the index, which is a requirement for being able to actually
- find a file that has been added to the repo and then deleted
- from the working tree.
-
Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
## Documentation/config/core.txt ##
@@ Documentation/config/core.txt: core.whitespace::
+ hardened via the core.fsyncMethod when created or modified. You can
+ disable hardening of any component by prefixing it with a '-'. Later
+ items take precedence over earlier ones in the list. For example,
-+ `core.fsync=all,-index` means "harden everything except the index".
-+ Items that are not hardened may be lost in the event of an unclean
-+ system shutdown.
++ `core.fsync=all,-pack-metadata` means "harden everything except pack
++ metadata." Items that are not hardened may be lost in the event of an
++ unclean system shutdown.
++
+* `none` disables fsync completely. This must be specified alone.
+* `loose-object` hardens objects added to the repo in loose-object form.
+* `pack` hardens objects added to the repo in packfile form.
+* `pack-metadata` hardens packfile bitmaps and indexes.
+* `commit-graph` hardens the commit graph file.
-+* `index` hardens the index when it is modified.
+* `objects` is an aggregate option that includes `loose-objects`, `pack`,
+ `pack-metadata`, and `commit-graph`.
+* `default` is an aggregate option that is equivalent to `objects,-loose-object`
@@ Documentation/config/core.txt: core.whitespace::
A value indicating the strategy Git will use to harden repository data
using fsync and related primitives.
@@ Documentation/config/core.txt: core.fsyncMethod::
- hardware, but does not send any FLUSH CACHE request. If the operating system
- does not support the required interfaces, this falls back to fsync().
+ filesystem and storage hardware, data added to the repository may not be
+ durable in the event of a system crash. This is the default mode on macOS.
-core.fsyncObjectFiles::
- This boolean will enable 'fsync()' when writing object files.
@@ builtin/fast-import.c: static void end_packfile(void)
close_pack_windows(pack_data);
- finalize_hashfile(pack_file, cur_pack_oid.hash, 0);
-+ finalize_hashfile(pack_file, cur_pack_oid.hash, REPO_COMPONENT_PACK, 0);
++ finalize_hashfile(pack_file, cur_pack_oid.hash, FSYNC_COMPONENT_PACK, 0);
fixup_pack_header_footer(pack_data->pack_fd, pack_data->hash,
pack_data->pack_name, object_count,
cur_pack_oid.hash, pack_size);
@@ builtin/index-pack.c: static void conclude_pack(int fix_thin_pack, const char *c
stop_progress_msg(&progress, msg.buf);
strbuf_release(&msg);
- finalize_hashfile(f, tail_hash, 0);
-+ finalize_hashfile(f, tail_hash, REPO_COMPONENT_PACK, 0);
++ finalize_hashfile(f, tail_hash, FSYNC_COMPONENT_PACK, 0);
hashcpy(read_hash, pack_hash);
fixup_pack_header_footer(output_fd, pack_hash,
curr_pack, nr_objects,
@@ builtin/index-pack.c: static void final(const char *final_pack_name, const char
close(input_fd);
} else {
- fsync_or_die(output_fd, curr_pack_name);
-+ fsync_component_or_die(REPO_COMPONENT_PACK, output_fd, curr_pack_name);
++ fsync_component_or_die(FSYNC_COMPONENT_PACK, output_fd, curr_pack_name);
err = close(output_fd);
if (err)
die_errno(_("error while closing pack file"));
@@ builtin/pack-objects.c: static void write_pack_file(void)
*/
if (pack_to_stdout) {
- finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_CLOSE);
-+ finalize_hashfile(f, hash, REPO_COMPONENT_NONE,
++ finalize_hashfile(f, hash, FSYNC_COMPONENT_NONE,
+ CSUM_HASH_IN_STREAM | CSUM_CLOSE);
} else if (nr_written == nr_remaining) {
- finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
-+ finalize_hashfile(f, hash, REPO_COMPONENT_PACK,
++ finalize_hashfile(f, hash, FSYNC_COMPONENT_PACK,
+ CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
} else {
- int fd = finalize_hashfile(f, hash, 0);
-+ int fd = finalize_hashfile(f, hash, REPO_COMPONENT_PACK, 0);
++ int fd = finalize_hashfile(f, hash, FSYNC_COMPONENT_PACK, 0);
fixup_pack_header_footer(fd, hash, pack_tmp_name,
nr_written, hash, offset);
close(fd);
@@ bulk-checkin.c: static void finish_bulk_checkin(struct bulk_checkin_state *state
goto clear_exit;
} else if (state->nr_written == 1) {
- finalize_hashfile(state->f, hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
-+ finalize_hashfile(state->f, hash, REPO_COMPONENT_PACK,
++ finalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK,
+ CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
} else {
- int fd = finalize_hashfile(state->f, hash, 0);
-+ int fd = finalize_hashfile(state->f, hash, REPO_COMPONENT_PACK, 0);
++ int fd = finalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK, 0);
fixup_pack_header_footer(fd, hash, state->pack_tmp_name,
state->nr_written, hash,
state->offset);
@@ cache.h: void reset_shared_repository(void);
-extern int fsync_object_files;
+/*
+ * These values are used to help identify parts of a repository to fsync.
-+ * REPO_COMPONENT_NONE identifies data that will not be a persistent part of the
++ * FSYNC_COMPONENT_NONE identifies data that will not be a persistent part of the
+ * repository and so shouldn't be fsynced.
+ */
-+enum repo_component {
-+ REPO_COMPONENT_NONE = 0,
-+ REPO_COMPONENT_LOOSE_OBJECT = 1 << 0,
-+ REPO_COMPONENT_PACK = 1 << 1,
-+ REPO_COMPONENT_PACK_METADATA = 1 << 2,
-+ REPO_COMPONENT_COMMIT_GRAPH = 1 << 3,
-+ REPO_COMPONENT_INDEX = 1 << 4,
++enum fsync_component {
++ FSYNC_COMPONENT_NONE = 0,
++ FSYNC_COMPONENT_LOOSE_OBJECT = 1 << 0,
++ FSYNC_COMPONENT_PACK = 1 << 1,
++ FSYNC_COMPONENT_PACK_METADATA = 1 << 2,
++ FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3,
+};
+
-+#define FSYNC_COMPONENTS_DEFAULT (REPO_COMPONENT_PACK | \
-+ REPO_COMPONENT_PACK_METADATA | \
-+ REPO_COMPONENT_COMMIT_GRAPH)
++#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \
++ FSYNC_COMPONENT_PACK_METADATA | \
++ FSYNC_COMPONENT_COMMIT_GRAPH)
+
-+#define FSYNC_COMPONENTS_OBJECTS (REPO_COMPONENT_LOOSE_OBJECT | \
-+ REPO_COMPONENT_PACK | \
-+ REPO_COMPONENT_PACK_METADATA | \
-+ REPO_COMPONENT_COMMIT_GRAPH)
++#define FSYNC_COMPONENTS_OBJECTS (FSYNC_COMPONENT_LOOSE_OBJECT | \
++ FSYNC_COMPONENT_PACK | \
++ FSYNC_COMPONENT_PACK_METADATA | \
++ FSYNC_COMPONENT_COMMIT_GRAPH)
+
-+#define FSYNC_COMPONENTS_ALL (REPO_COMPONENT_LOOSE_OBJECT | \
-+ REPO_COMPONENT_PACK | \
-+ REPO_COMPONENT_PACK_METADATA | \
-+ REPO_COMPONENT_COMMIT_GRAPH | \
-+ REPO_COMPONENT_INDEX)
++#define FSYNC_COMPONENTS_ALL (FSYNC_COMPONENT_LOOSE_OBJECT | \
++ FSYNC_COMPONENT_PACK | \
++ FSYNC_COMPONENT_PACK_METADATA | \
++ FSYNC_COMPONENT_COMMIT_GRAPH)
+
+
+/*
+ * A bitmask indicating which components of the repo should be fsynced.
+ */
-+extern enum repo_component fsync_components;
++extern enum fsync_component fsync_components;
enum fsync_method {
FSYNC_METHOD_FSYNC,
@@ cache.h: int copy_file_with_time(const char *dst, const char *src, int mode);
void write_or_die(int fd, const void *buf, size_t count);
void fsync_or_die(int fd, const char *);
-+inline void fsync_component_or_die(enum repo_component component, int fd, const char *msg)
++inline void fsync_component_or_die(enum fsync_component component, int fd, const char *msg)
+{
+ if (fsync_components & component)
+ fsync_or_die(fd, msg);
@@ commit-graph.c: static int write_commit_graph_file(struct write_commit_graph_con
close_commit_graph(ctx->r->objects);
- finalize_hashfile(f, file_hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC);
-+ finalize_hashfile(f, file_hash, REPO_COMPONENT_COMMIT_GRAPH,
++ finalize_hashfile(f, file_hash, FSYNC_COMPONENT_COMMIT_GRAPH,
+ CSUM_HASH_IN_STREAM | CSUM_FSYNC);
free_chunkfile(cf);
@@ config.c: static int git_parse_maybe_bool_text(const char *value)
+static const struct fsync_component_entry {
+ const char *name;
-+ enum repo_component component_bits;
++ enum fsync_component component_bits;
+} fsync_component_table[] = {
-+ { "loose-object", REPO_COMPONENT_LOOSE_OBJECT },
-+ { "pack", REPO_COMPONENT_PACK },
-+ { "pack-metadata", REPO_COMPONENT_PACK_METADATA },
-+ { "commit-graph", REPO_COMPONENT_COMMIT_GRAPH },
-+ { "index", REPO_COMPONENT_INDEX },
++ { "loose-object", FSYNC_COMPONENT_LOOSE_OBJECT },
++ { "pack", FSYNC_COMPONENT_PACK },
++ { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA },
++ { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH },
+ { "objects", FSYNC_COMPONENTS_OBJECTS },
+ { "default", FSYNC_COMPONENTS_DEFAULT },
+ { "all", FSYNC_COMPONENTS_ALL },
+};
+
-+static enum repo_component parse_fsync_components(const char *var, const char *string)
++static enum fsync_component parse_fsync_components(const char *var, const char *string)
+{
-+ enum repo_component output = 0;
++ enum fsync_component output = 0;
+
+ if (!strcmp(string, "none"))
+ return output;
@@ csum-file.c: static void free_hashfile(struct hashfile *f)
-int finalize_hashfile(struct hashfile *f, unsigned char *result, unsigned int flags)
+int finalize_hashfile(struct hashfile *f, unsigned char *result,
-+ enum repo_component component, unsigned int flags)
++ enum fsync_component component, unsigned int flags)
{
int fd;
@@ csum-file.c: int finalize_hashfile(struct hashfile *f, unsigned char *result, un
die_errno("%s: sha1 file error on close", f->name);
## csum-file.h ##
+@@
+ #ifndef CSUM_FILE_H
+ #define CSUM_FILE_H
+
++#include "cache.h"
+ #include "hash.h"
+
+ struct progress;
@@ csum-file.h: int hashfile_truncate(struct hashfile *, struct hashfile_checkpoint *);
struct hashfile *hashfd(int fd, const char *name);
struct hashfile *hashfd_check(const char *name);
struct hashfile *hashfd_throughput(int fd, const char *name, struct progress *tp);
-int finalize_hashfile(struct hashfile *, unsigned char *, unsigned int);
-+int finalize_hashfile(struct hashfile *, unsigned char *, enum repo_component, unsigned int);
++int finalize_hashfile(struct hashfile *, unsigned char *, enum fsync_component, unsigned int);
void hashwrite(struct hashfile *, const void *, unsigned int);
void hashflush(struct hashfile *f);
void crc32_begin(struct hashfile *);
@@ environment.c: const char *git_hooks_path;
int zlib_compression_level = Z_BEST_SPEED;
int pack_compression_level = Z_DEFAULT_COMPRESSION;
enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT;
-+enum repo_component fsync_components = FSYNC_COMPONENTS_DEFAULT;
++enum fsync_component fsync_components = FSYNC_COMPONENTS_DEFAULT;
size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE;
size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT;
size_t delta_base_cache_limit = 96 * 1024 * 1024;
@@ midx.c: static int write_midx_internal(const char *object_dir,
write_chunkfile(cf, &ctx);
- finalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);
-+ finalize_hashfile(f, midx_hash, REPO_COMPONENT_PACK_METADATA,
++ finalize_hashfile(f, midx_hash, FSYNC_COMPONENT_PACK_METADATA,
+ CSUM_FSYNC | CSUM_HASH_IN_STREAM);
free_chunkfile(cf);
@@ object-file.c: int hash_object_file(const struct git_hash_algo *algo, const void
{
- if (fsync_object_files)
- fsync_or_die(fd, "loose object file");
-+ fsync_component_or_die(REPO_COMPONENT_LOOSE_OBJECT, fd, "loose object file");
++ fsync_component_or_die(FSYNC_COMPONENT_LOOSE_OBJECT, fd, "loose object file");
if (close(fd) != 0)
die_errno(_("error when closing loose object file"));
}
@@ pack-bitmap-write.c: void bitmap_writer_finish(struct pack_idx_entry **index,
write_hash_cache(f, index, index_nr);
- finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
-+ finalize_hashfile(f, NULL, REPO_COMPONENT_PACK_METADATA,
++ finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA,
+ CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
if (adjust_shared_perm(tmp_file.buf))
@@ pack-write.c: const char *write_idx_file(const char *index_name, struct pack_idx
hashwrite(f, sha1, the_hash_algo->rawsz);
- finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE |
- ((opts->flags & WRITE_IDX_VERIFY)
-+ finalize_hashfile(f, NULL, REPO_COMPONENT_PACK_METADATA,
+- ? 0 : CSUM_FSYNC));
++ finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA,
+ CSUM_HASH_IN_STREAM | CSUM_CLOSE |
-+ ((opts->flags & WRITE_IDX_VERIFY)
- ? 0 : CSUM_FSYNC));
++ ((opts->flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC));
return index_name;
}
+
@@ pack-write.c: const char *write_rev_file_order(const char *rev_name,
if (rev_name && adjust_shared_perm(rev_name) < 0)
die(_("failed to make %s readable"), rev_name);
- finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE |
- ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC));
-+ finalize_hashfile(f, NULL, REPO_COMPONENT_PACK_METADATA,
++ finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA,
+ CSUM_HASH_IN_STREAM | CSUM_CLOSE |
+ ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC));
@@ pack-write.c: void fixup_pack_header_footer(int pack_fd,
the_hash_algo->final_fn(new_pack_hash, &new_hash_ctx);
write_or_die(pack_fd, new_pack_hash, the_hash_algo->rawsz);
- fsync_or_die(pack_fd, pack_name);
-+ fsync_component_or_die(REPO_COMPONENT_PACK, pack_fd, pack_name);
++ fsync_component_or_die(FSYNC_COMPONENT_PACK, pack_fd, pack_name);
}
char *index_pack_lockfile(int ip_out, int *is_well_formed)
## read-cache.c ##
-@@ read-cache.c: static int record_ieot(void)
- * rely on it.
- */
- static int do_write_index(struct index_state *istate, struct tempfile *tempfile,
-- int strip_extensions)
-+ int strip_extensions, unsigned flags)
- {
- uint64_t start = getnanotime();
- struct hashfile *f;
-@@ read-cache.c: static int do_write_index(struct index_state *istate, struct tempfile *tempfile,
- struct strbuf previous_name_buf = STRBUF_INIT, *previous_name;
- int drop_cache_tree = istate->drop_cache_tree;
- off_t offset;
-+ int csum_fsync_flag;
- int ieot_entries = 1;
- struct index_entry_offset_table *ieot = NULL;
- int nr, nr_threads;
@@ read-cache.c: static int do_write_index(struct index_state *istate, struct tempfile *tempfile,
return -1;
}
- finalize_hashfile(f, istate->oid.hash, CSUM_HASH_IN_STREAM);
-+ csum_fsync_flag = 0;
-+ if (!alternate_index_output && (flags & COMMIT_LOCK))
-+ csum_fsync_flag = CSUM_FSYNC;
-+
-+ finalize_hashfile(f, istate->oid.hash, REPO_COMPONENT_INDEX,
-+ CSUM_HASH_IN_STREAM | csum_fsync_flag);
-+
++ finalize_hashfile(f, istate->oid.hash, FSYNC_COMPONENT_NONE, CSUM_HASH_IN_STREAM);
if (close_tempfile_gently(tempfile)) {
error(_("could not close '%s'"), get_tempfile_path(tempfile));
return -1;
-@@ read-cache.c: static int do_write_locked_index(struct index_state *istate, struct lock_file *l
- */
- trace2_region_enter_printf("index", "do_write_index", the_repository,
- "%s", get_lock_file_path(lock));
-- ret = do_write_index(istate, lock->tempfile, 0);
-+ ret = do_write_index(istate, lock->tempfile, 0, flags);
- trace2_region_leave_printf("index", "do_write_index", the_repository,
- "%s", get_lock_file_path(lock));
-
-@@ read-cache.c: static int clean_shared_index_files(const char *current_hex)
- }
-
- static int write_shared_index(struct index_state *istate,
-- struct tempfile **temp)
-+ struct tempfile **temp, unsigned flags)
- {
- struct split_index *si = istate->split_index;
- int ret, was_full = !istate->sparse_index;
-@@ read-cache.c: static int write_shared_index(struct index_state *istate,
-
- trace2_region_enter_printf("index", "shared/do_write_index",
- the_repository, "%s", get_tempfile_path(*temp));
-- ret = do_write_index(si->base, *temp, 1);
-+ ret = do_write_index(si->base, *temp, 1, flags);
- trace2_region_leave_printf("index", "shared/do_write_index",
- the_repository, "%s", get_tempfile_path(*temp));
-
-@@ read-cache.c: int write_locked_index(struct index_state *istate, struct lock_file *lock,
- ret = do_write_locked_index(istate, lock, flags);
- goto out;
- }
-- ret = write_shared_index(istate, &temp);
-+ ret = write_shared_index(istate, &temp, flags);
-
- saved_errno = errno;
- if (is_tempfile_active(temp))
-: ----------- > 3: 86e39b8f8d1 core.fsync: new option to harden the index
--
gitgitgadget
^ permalink raw reply [flat|nested] 122+ messages in thread
* [PATCH v2 1/3] core.fsyncmethod: add writeout-only mode
2021-12-07 2:46 ` [PATCH v2 0/3] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget
@ 2021-12-07 2:46 ` Neeraj Singh via GitGitGadget
2021-12-07 11:44 ` Patrick Steinhardt
2021-12-07 12:18 ` Ævar Arnfjörð Bjarmason
2021-12-07 2:46 ` [PATCH v2 2/3] core.fsync: introduce granular fsync control Neeraj Singh via GitGitGadget
` (3 subsequent siblings)
4 siblings, 2 replies; 122+ messages in thread
From: Neeraj Singh via GitGitGadget @ 2021-12-07 2:46 UTC (permalink / raw)
To: git
Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps,
Neeraj K. Singh, Neeraj Singh
From: Neeraj Singh <neerajsi@microsoft.com>
This commit introduces the `core.fsyncmethod` configuration
knob, which can currently be set to `fsync` or `writeout-only`.
The new writeout-only mode attempts to tell the operating system to
flush its in-memory page cache to the storage hardware without issuing a
CACHE_FLUSH command to the storage controller.
Writeout-only fsync is significantly faster than a vanilla fsync on
common hardware, since data is written to a disk-side cache rather than
all the way to a durable medium. Later changes in this patch series will
take advantage of this primitive to implement batching of hardware
flushes.
When git_fsync is called with FSYNC_WRITEOUT_ONLY, it may fail and the
caller is expected to do an ordinary fsync as needed.
On Apple platforms, the fsync system call does not issue a CACHE_FLUSH
directive to the storage controller. This change updates fsync to do
fcntl(F_FULLFSYNC) to make fsync actually durable. We maintain parity
with existing behavior on Apple platforms by setting the default value
of the new core.fsyncmethod option.
Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
---
Documentation/config/core.txt | 9 +++++
Makefile | 6 ++++
cache.h | 7 ++++
compat/mingw.h | 3 ++
compat/win32/flush.c | 28 +++++++++++++++
config.c | 12 +++++++
config.mak.uname | 3 ++
configure.ac | 8 +++++
contrib/buildsystems/CMakeLists.txt | 3 +-
environment.c | 2 +-
git-compat-util.h | 24 +++++++++++++
wrapper.c | 56 +++++++++++++++++++++++++++++
write-or-die.c | 10 +++---
13 files changed, 165 insertions(+), 6 deletions(-)
create mode 100644 compat/win32/flush.c
diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
index c04f62a54a1..dbb134f7136 100644
--- a/Documentation/config/core.txt
+++ b/Documentation/config/core.txt
@@ -547,6 +547,15 @@ core.whitespace::
is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent`
errors. The default tab width is 8. Allowed values are 1 to 63.
+core.fsyncMethod::
+ A value indicating the strategy Git will use to harden repository data
+ using fsync and related primitives.
++
+* `fsync` uses the fsync() system call or platform equivalents.
+* `writeout-only` issues pagecache writeback requests, but depending on the
+ filesystem and storage hardware, data added to the repository may not be
+ durable in the event of a system crash. This is the default mode on macOS.
+
core.fsyncObjectFiles::
This boolean will enable 'fsync()' when writing object files.
+
diff --git a/Makefile b/Makefile
index d56c0e4aadc..cba024615c9 100644
--- a/Makefile
+++ b/Makefile
@@ -403,6 +403,8 @@ all::
#
# Define HAVE_CLOCK_MONOTONIC if your platform has CLOCK_MONOTONIC.
#
+# Define HAVE_SYNC_FILE_RANGE if your platform has sync_file_range.
+#
# Define NEEDS_LIBRT if your platform requires linking with librt (glibc version
# before 2.17) for clock_gettime and CLOCK_MONOTONIC.
#
@@ -1881,6 +1883,10 @@ ifdef HAVE_CLOCK_MONOTONIC
BASIC_CFLAGS += -DHAVE_CLOCK_MONOTONIC
endif
+ifdef HAVE_SYNC_FILE_RANGE
+ BASIC_CFLAGS += -DHAVE_SYNC_FILE_RANGE
+endif
+
ifdef NEEDS_LIBRT
EXTLIBS += -lrt
endif
diff --git a/cache.h b/cache.h
index eba12487b99..9cd60d94952 100644
--- a/cache.h
+++ b/cache.h
@@ -986,6 +986,13 @@ extern int read_replace_refs;
extern char *git_replace_ref_base;
extern int fsync_object_files;
+
+enum fsync_method {
+ FSYNC_METHOD_FSYNC,
+ FSYNC_METHOD_WRITEOUT_ONLY
+};
+
+extern enum fsync_method fsync_method;
extern int core_preload_index;
extern int precomposed_unicode;
extern int protect_hfs;
diff --git a/compat/mingw.h b/compat/mingw.h
index c9a52ad64a6..6074a3d3ced 100644
--- a/compat/mingw.h
+++ b/compat/mingw.h
@@ -329,6 +329,9 @@ int mingw_getpagesize(void);
#define getpagesize mingw_getpagesize
#endif
+int win32_fsync_no_flush(int fd);
+#define fsync_no_flush win32_fsync_no_flush
+
struct rlimit {
unsigned int rlim_cur;
};
diff --git a/compat/win32/flush.c b/compat/win32/flush.c
new file mode 100644
index 00000000000..75324c24ee7
--- /dev/null
+++ b/compat/win32/flush.c
@@ -0,0 +1,28 @@
+#include "../../git-compat-util.h"
+#include <winternl.h>
+#include "lazyload.h"
+
+int win32_fsync_no_flush(int fd)
+{
+ IO_STATUS_BLOCK io_status;
+
+#define FLUSH_FLAGS_FILE_DATA_ONLY 1
+
+ DECLARE_PROC_ADDR(ntdll.dll, NTSTATUS, NtFlushBuffersFileEx,
+ HANDLE FileHandle, ULONG Flags, PVOID Parameters, ULONG ParameterSize,
+ PIO_STATUS_BLOCK IoStatusBlock);
+
+ if (!INIT_PROC_ADDR(NtFlushBuffersFileEx)) {
+ errno = ENOSYS;
+ return -1;
+ }
+
+ memset(&io_status, 0, sizeof(io_status));
+ if (NtFlushBuffersFileEx((HANDLE)_get_osfhandle(fd), FLUSH_FLAGS_FILE_DATA_ONLY,
+ NULL, 0, &io_status)) {
+ errno = EINVAL;
+ return -1;
+ }
+
+ return 0;
+}
diff --git a/config.c b/config.c
index c5873f3a706..c3410b8a868 100644
--- a/config.c
+++ b/config.c
@@ -1490,6 +1490,18 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
return 0;
}
+ if (!strcmp(var, "core.fsyncmethod")) {
+ if (!value)
+ return config_error_nonbool(var);
+ if (!strcmp(value, "fsync"))
+ fsync_method = FSYNC_METHOD_FSYNC;
+ else if (!strcmp(value, "writeout-only"))
+ fsync_method = FSYNC_METHOD_WRITEOUT_ONLY;
+ else
+ warning(_("unknown %s value '%s'"), var, value);
+
+ }
+
if (!strcmp(var, "core.fsyncobjectfiles")) {
fsync_object_files = git_config_bool(var, value);
return 0;
diff --git a/config.mak.uname b/config.mak.uname
index d0701f9beb0..774a09622d2 100644
--- a/config.mak.uname
+++ b/config.mak.uname
@@ -57,6 +57,7 @@ ifeq ($(uname_S),Linux)
HAVE_CLOCK_MONOTONIC = YesPlease
# -lrt is needed for clock_gettime on glibc <= 2.16
NEEDS_LIBRT = YesPlease
+ HAVE_SYNC_FILE_RANGE = YesPlease
HAVE_GETDELIM = YesPlease
FREAD_READS_DIRECTORIES = UnfortunatelyYes
BASIC_CFLAGS += -DHAVE_SYSINFO
@@ -453,6 +454,7 @@ endif
CFLAGS =
BASIC_CFLAGS = -nologo -I. -Icompat/vcbuild/include -DWIN32 -D_CONSOLE -DHAVE_STRING_H -D_CRT_SECURE_NO_WARNINGS -D_CRT_NONSTDC_NO_DEPRECATE
COMPAT_OBJS = compat/msvc.o compat/winansi.o \
+ compat/win32/flush.o \
compat/win32/path-utils.o \
compat/win32/pthread.o compat/win32/syslog.o \
compat/win32/trace2_win32_process_info.o \
@@ -628,6 +630,7 @@ ifeq ($(uname_S),MINGW)
COMPAT_CFLAGS += -DSTRIP_EXTENSION=\".exe\"
COMPAT_OBJS += compat/mingw.o compat/winansi.o \
compat/win32/trace2_win32_process_info.o \
+ compat/win32/flush.o \
compat/win32/path-utils.o \
compat/win32/pthread.o compat/win32/syslog.o \
compat/win32/dirent.o
diff --git a/configure.ac b/configure.ac
index 5ee25ec95c8..6bd6bef1c44 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1082,6 +1082,14 @@ AC_COMPILE_IFELSE([CLOCK_MONOTONIC_SRC],
[AC_MSG_RESULT([no])
HAVE_CLOCK_MONOTONIC=])
GIT_CONF_SUBST([HAVE_CLOCK_MONOTONIC])
+
+#
+# Define HAVE_SYNC_FILE_RANGE=YesPlease if sync_file_range is available.
+GIT_CHECK_FUNC(sync_file_range,
+ [HAVE_SYNC_FILE_RANGE=YesPlease],
+ [HAVE_SYNC_FILE_RANGE])
+GIT_CONF_SUBST([HAVE_SYNC_FILE_RANGE])
+
#
# Define NO_SETITIMER if you don't have setitimer.
GIT_CHECK_FUNC(setitimer,
diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt
index 86b46114464..6d7bc16d054 100644
--- a/contrib/buildsystems/CMakeLists.txt
+++ b/contrib/buildsystems/CMakeLists.txt
@@ -261,7 +261,8 @@ if(CMAKE_SYSTEM_NAME STREQUAL "Windows")
NOGDI OBJECT_CREATION_MODE=1 __USE_MINGW_ANSI_STDIO=0
USE_NED_ALLOCATOR OVERRIDE_STRDUP MMAP_PREVENTS_DELETE USE_WIN32_MMAP
UNICODE _UNICODE HAVE_WPGMPTR ENSURE_MSYSTEM_IS_SET)
- list(APPEND compat_SOURCES compat/mingw.c compat/winansi.c compat/win32/path-utils.c
+ list(APPEND compat_SOURCES compat/mingw.c compat/winansi.c
+ compat/win32/flush.c compat/win32/path-utils.c
compat/win32/pthread.c compat/win32mmap.c compat/win32/syslog.c
compat/win32/trace2_win32_process_info.c compat/win32/dirent.c
compat/nedmalloc/nedmalloc.c compat/strdup.c)
diff --git a/environment.c b/environment.c
index 9da7f3c1a19..f9140e842cf 100644
--- a/environment.c
+++ b/environment.c
@@ -41,7 +41,7 @@ const char *git_attributes_file;
const char *git_hooks_path;
int zlib_compression_level = Z_BEST_SPEED;
int pack_compression_level = Z_DEFAULT_COMPRESSION;
-int fsync_object_files;
+enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT;
size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE;
size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT;
size_t delta_base_cache_limit = 96 * 1024 * 1024;
diff --git a/git-compat-util.h b/git-compat-util.h
index c6bd2a84e55..cb9abd7a08c 100644
--- a/git-compat-util.h
+++ b/git-compat-util.h
@@ -1239,6 +1239,30 @@ __attribute__((format (printf, 1, 2))) NORETURN
void BUG(const char *fmt, ...);
#endif
+#ifdef __APPLE__
+#define FSYNC_METHOD_DEFAULT FSYNC_METHOD_WRITEOUT_ONLY
+#else
+#define FSYNC_METHOD_DEFAULT FSYNC_METHOD_FSYNC
+#endif
+
+enum fsync_action {
+ FSYNC_WRITEOUT_ONLY,
+ FSYNC_HARDWARE_FLUSH
+};
+
+/*
+ * Issues an fsync against the specified file according to the specified mode.
+ *
+ * FSYNC_WRITEOUT_ONLY attempts to use interfaces available on some operating
+ * systems to flush the OS cache without issuing a flush command to the storage
+ * controller. If those interfaces are unavailable, the function fails with
+ * ENOSYS.
+ *
+ * FSYNC_HARDWARE_FLUSH does an OS writeout and hardware flush to ensure that
+ * changes are durable. It is not expected to fail.
+ */
+int git_fsync(int fd, enum fsync_action action);
+
/*
* Preserves errno, prints a message, but gives no warning for ENOENT.
* Returns 0 on success, which includes trying to unlink an object that does
diff --git a/wrapper.c b/wrapper.c
index 36e12119d76..1c5f2c87791 100644
--- a/wrapper.c
+++ b/wrapper.c
@@ -546,6 +546,62 @@ int xmkstemp_mode(char *filename_template, int mode)
return fd;
}
+int git_fsync(int fd, enum fsync_action action)
+{
+ switch (action) {
+ case FSYNC_WRITEOUT_ONLY:
+
+#ifdef __APPLE__
+ /*
+ * on macOS, fsync just causes filesystem cache writeback but does not
+ * flush hardware caches.
+ */
+ return fsync(fd);
+#endif
+
+#ifdef HAVE_SYNC_FILE_RANGE
+ /*
+ * On linux 2.6.17 and above, sync_file_range is the way to issue
+ * a writeback without a hardware flush. An offset of 0 and size of 0
+ * indicates writeout of the entire file and the wait flags ensure that all
+ * dirty data is written to the disk (potentially in a disk-side cache)
+ * before we continue.
+ */
+
+ return sync_file_range(fd, 0, 0, SYNC_FILE_RANGE_WAIT_BEFORE |
+ SYNC_FILE_RANGE_WRITE |
+ SYNC_FILE_RANGE_WAIT_AFTER);
+#endif
+
+#ifdef fsync_no_flush
+ return fsync_no_flush(fd);
+#endif
+
+ errno = ENOSYS;
+ return -1;
+
+ case FSYNC_HARDWARE_FLUSH:
+ /*
+ * On some platforms fsync may return EINTR. Try again in this
+ * case, since callers asking for a hardware flush may die if
+ * this function returns an error.
+ */
+ for (;;) {
+ int err;
+#ifdef __APPLE__
+ err = fcntl(fd, F_FULLFSYNC);
+#else
+ err = fsync(fd);
+#endif
+ if (err >= 0 || errno != EINTR)
+ return err;
+ }
+
+ default:
+ BUG("unexpected git_fsync(%d) call", action);
+ }
+}
+
static int warn_if_unremovable(const char *op, const char *file, int rc)
{
int err;
diff --git a/write-or-die.c b/write-or-die.c
index 0b1ec8190b6..0702acdd5e8 100644
--- a/write-or-die.c
+++ b/write-or-die.c
@@ -57,10 +57,12 @@ void fprintf_or_die(FILE *f, const char *fmt, ...)
void fsync_or_die(int fd, const char *msg)
{
- while (fsync(fd) < 0) {
- if (errno != EINTR)
- die_errno("fsync error on '%s'", msg);
- }
+ if (fsync_method == FSYNC_METHOD_WRITEOUT_ONLY &&
+ git_fsync(fd, FSYNC_WRITEOUT_ONLY) >= 0)
+ return;
+
+ if (git_fsync(fd, FSYNC_HARDWARE_FLUSH) < 0)
+ die_errno("fsync error on '%s'", msg);
}
void write_or_die(int fd, const void *buf, size_t count)
--
gitgitgadget
^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v2 2/3] core.fsync: introduce granular fsync control
2021-12-07 2:46 ` [PATCH v2 0/3] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget
2021-12-07 2:46 ` [PATCH v2 1/3] core.fsyncmethod: add writeout-only mode Neeraj Singh via GitGitGadget
@ 2021-12-07 2:46 ` Neeraj Singh via GitGitGadget
2021-12-07 11:53 ` Patrick Steinhardt
2021-12-07 12:29 ` Ævar Arnfjörð Bjarmason
2021-12-07 2:46 ` [PATCH v2 3/3] core.fsync: new option to harden the index Neeraj Singh via GitGitGadget
` (2 subsequent siblings)
4 siblings, 2 replies; 122+ messages in thread
From: Neeraj Singh via GitGitGadget @ 2021-12-07 2:46 UTC (permalink / raw)
To: git
Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps,
Neeraj K. Singh, Neeraj Singh
From: Neeraj Singh <neerajsi@microsoft.com>
This commit introduces the `core.fsync` configuration
knob which can be used to control how components of the
repository are made durable on disk.
This setting allows future extensibility of the list of
syncable components:
* We issue a warning rather than an error for unrecognized
components, so new configs can be used with old Git versions.
* We support negation, so users can choose one of the default
aggregate options and then remove components that they don't
want. The user would then harden any new components added in
a Git version update.
This also supports the common request of doing absolutely no
fysncing with the `core.fsync=none` value, which is expected
to make the test suite faster.
Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
---
Documentation/config/core.txt | 27 +++++++++----
builtin/fast-import.c | 2 +-
builtin/index-pack.c | 4 +-
builtin/pack-objects.c | 8 ++--
bulk-checkin.c | 5 ++-
cache.h | 39 +++++++++++++++++-
commit-graph.c | 3 +-
config.c | 76 ++++++++++++++++++++++++++++++++++-
csum-file.c | 5 ++-
csum-file.h | 3 +-
environment.c | 1 +
midx.c | 3 +-
object-file.c | 3 +-
pack-bitmap-write.c | 3 +-
pack-write.c | 13 +++---
read-cache.c | 2 +-
16 files changed, 164 insertions(+), 33 deletions(-)
diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
index dbb134f7136..4f1747ec871 100644
--- a/Documentation/config/core.txt
+++ b/Documentation/config/core.txt
@@ -547,6 +547,25 @@ core.whitespace::
is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent`
errors. The default tab width is 8. Allowed values are 1 to 63.
+core.fsync::
+ A comma-separated list of parts of the repository which should be
+ hardened via the core.fsyncMethod when created or modified. You can
+ disable hardening of any component by prefixing it with a '-'. Later
+ items take precedence over earlier ones in the list. For example,
+ `core.fsync=all,-pack-metadata` means "harden everything except pack
+ metadata." Items that are not hardened may be lost in the event of an
+ unclean system shutdown.
++
+* `none` disables fsync completely. This must be specified alone.
+* `loose-object` hardens objects added to the repo in loose-object form.
+* `pack` hardens objects added to the repo in packfile form.
+* `pack-metadata` hardens packfile bitmaps and indexes.
+* `commit-graph` hardens the commit graph file.
+* `objects` is an aggregate option that includes `loose-objects`, `pack`,
+ `pack-metadata`, and `commit-graph`.
+* `default` is an aggregate option that is equivalent to `objects,-loose-object`
+* `all` is an aggregate option that syncs all individual components above.
+
core.fsyncMethod::
A value indicating the strategy Git will use to harden repository data
using fsync and related primitives.
@@ -556,14 +575,6 @@ core.fsyncMethod::
filesystem and storage hardware, data added to the repository may not be
durable in the event of a system crash. This is the default mode on macOS.
-core.fsyncObjectFiles::
- This boolean will enable 'fsync()' when writing object files.
-+
-This is a total waste of time and effort on a filesystem that orders
-data writes properly, but can be useful for filesystems that do not use
-journalling (traditional UNIX filesystems) or that only journal metadata
-and not file contents (OS X's HFS+, or Linux ext3 with "data=writeback").
-
core.preloadIndex::
Enable parallel index preload for operations like 'git diff'
+
diff --git a/builtin/fast-import.c b/builtin/fast-import.c
index 20406f67754..e27a4580f85 100644
--- a/builtin/fast-import.c
+++ b/builtin/fast-import.c
@@ -856,7 +856,7 @@ static void end_packfile(void)
struct tag *t;
close_pack_windows(pack_data);
- finalize_hashfile(pack_file, cur_pack_oid.hash, 0);
+ finalize_hashfile(pack_file, cur_pack_oid.hash, FSYNC_COMPONENT_PACK, 0);
fixup_pack_header_footer(pack_data->pack_fd, pack_data->hash,
pack_data->pack_name, object_count,
cur_pack_oid.hash, pack_size);
diff --git a/builtin/index-pack.c b/builtin/index-pack.c
index c23d01de7dc..c32534c13b4 100644
--- a/builtin/index-pack.c
+++ b/builtin/index-pack.c
@@ -1286,7 +1286,7 @@ static void conclude_pack(int fix_thin_pack, const char *curr_pack, unsigned cha
nr_objects - nr_objects_initial);
stop_progress_msg(&progress, msg.buf);
strbuf_release(&msg);
- finalize_hashfile(f, tail_hash, 0);
+ finalize_hashfile(f, tail_hash, FSYNC_COMPONENT_PACK, 0);
hashcpy(read_hash, pack_hash);
fixup_pack_header_footer(output_fd, pack_hash,
curr_pack, nr_objects,
@@ -1508,7 +1508,7 @@ static void final(const char *final_pack_name, const char *curr_pack_name,
if (!from_stdin) {
close(input_fd);
} else {
- fsync_or_die(output_fd, curr_pack_name);
+ fsync_component_or_die(FSYNC_COMPONENT_PACK, output_fd, curr_pack_name);
err = close(output_fd);
if (err)
die_errno(_("error while closing pack file"));
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index 857be7826f3..916c55d6ce9 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -1204,11 +1204,13 @@ static void write_pack_file(void)
* If so, rewrite it like in fast-import
*/
if (pack_to_stdout) {
- finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_CLOSE);
+ finalize_hashfile(f, hash, FSYNC_COMPONENT_NONE,
+ CSUM_HASH_IN_STREAM | CSUM_CLOSE);
} else if (nr_written == nr_remaining) {
- finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
+ finalize_hashfile(f, hash, FSYNC_COMPONENT_PACK,
+ CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
} else {
- int fd = finalize_hashfile(f, hash, 0);
+ int fd = finalize_hashfile(f, hash, FSYNC_COMPONENT_PACK, 0);
fixup_pack_header_footer(fd, hash, pack_tmp_name,
nr_written, hash, offset);
close(fd);
diff --git a/bulk-checkin.c b/bulk-checkin.c
index 8785b2ac806..a2cf9dcbc8d 100644
--- a/bulk-checkin.c
+++ b/bulk-checkin.c
@@ -53,9 +53,10 @@ static void finish_bulk_checkin(struct bulk_checkin_state *state)
unlink(state->pack_tmp_name);
goto clear_exit;
} else if (state->nr_written == 1) {
- finalize_hashfile(state->f, hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
+ finalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK,
+ CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
} else {
- int fd = finalize_hashfile(state->f, hash, 0);
+ int fd = finalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK, 0);
fixup_pack_header_footer(fd, hash, state->pack_tmp_name,
state->nr_written, hash,
state->offset);
diff --git a/cache.h b/cache.h
index 9cd60d94952..d83fbaf2619 100644
--- a/cache.h
+++ b/cache.h
@@ -985,7 +985,38 @@ void reset_shared_repository(void);
extern int read_replace_refs;
extern char *git_replace_ref_base;
-extern int fsync_object_files;
+/*
+ * These values are used to help identify parts of a repository to fsync.
+ * FSYNC_COMPONENT_NONE identifies data that will not be a persistent part of the
+ * repository and so shouldn't be fsynced.
+ */
+enum fsync_component {
+ FSYNC_COMPONENT_NONE = 0,
+ FSYNC_COMPONENT_LOOSE_OBJECT = 1 << 0,
+ FSYNC_COMPONENT_PACK = 1 << 1,
+ FSYNC_COMPONENT_PACK_METADATA = 1 << 2,
+ FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3,
+};
+
+#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \
+ FSYNC_COMPONENT_PACK_METADATA | \
+ FSYNC_COMPONENT_COMMIT_GRAPH)
+
+#define FSYNC_COMPONENTS_OBJECTS (FSYNC_COMPONENT_LOOSE_OBJECT | \
+ FSYNC_COMPONENT_PACK | \
+ FSYNC_COMPONENT_PACK_METADATA | \
+ FSYNC_COMPONENT_COMMIT_GRAPH)
+
+#define FSYNC_COMPONENTS_ALL (FSYNC_COMPONENT_LOOSE_OBJECT | \
+ FSYNC_COMPONENT_PACK | \
+ FSYNC_COMPONENT_PACK_METADATA | \
+ FSYNC_COMPONENT_COMMIT_GRAPH)
+
+
+/*
+ * A bitmask indicating which components of the repo should be fsynced.
+ */
+extern enum fsync_component fsync_components;
enum fsync_method {
FSYNC_METHOD_FSYNC,
@@ -1747,6 +1778,12 @@ int copy_file_with_time(const char *dst, const char *src, int mode);
void write_or_die(int fd, const void *buf, size_t count);
void fsync_or_die(int fd, const char *);
+inline void fsync_component_or_die(enum fsync_component component, int fd, const char *msg)
+{
+ if (fsync_components & component)
+ fsync_or_die(fd, msg);
+}
+
ssize_t read_in_full(int fd, void *buf, size_t count);
ssize_t write_in_full(int fd, const void *buf, size_t count);
ssize_t pread_in_full(int fd, void *buf, size_t count, off_t offset);
diff --git a/commit-graph.c b/commit-graph.c
index 2706683acfe..c8a5dea4541 100644
--- a/commit-graph.c
+++ b/commit-graph.c
@@ -1939,7 +1939,8 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)
}
close_commit_graph(ctx->r->objects);
- finalize_hashfile(f, file_hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC);
+ finalize_hashfile(f, file_hash, FSYNC_COMPONENT_COMMIT_GRAPH,
+ CSUM_HASH_IN_STREAM | CSUM_FSYNC);
free_chunkfile(cf);
if (ctx->split) {
diff --git a/config.c b/config.c
index c3410b8a868..29c867aab03 100644
--- a/config.c
+++ b/config.c
@@ -1213,6 +1213,73 @@ static int git_parse_maybe_bool_text(const char *value)
return -1;
}
+static const struct fsync_component_entry {
+ const char *name;
+ enum fsync_component component_bits;
+} fsync_component_table[] = {
+ { "loose-object", FSYNC_COMPONENT_LOOSE_OBJECT },
+ { "pack", FSYNC_COMPONENT_PACK },
+ { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA },
+ { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH },
+ { "objects", FSYNC_COMPONENTS_OBJECTS },
+ { "default", FSYNC_COMPONENTS_DEFAULT },
+ { "all", FSYNC_COMPONENTS_ALL },
+};
+
+static enum fsync_component parse_fsync_components(const char *var, const char *string)
+{
+ enum fsync_component output = 0;
+
+ if (!strcmp(string, "none"))
+ return output;
+
+ while (string) {
+ int i;
+ size_t len;
+ const char *ep;
+ int negated = 0;
+ int found = 0;
+
+ string = string + strspn(string, ", \t\n\r");
+ ep = strchrnul(string, ',');
+ len = ep - string;
+
+ if (*string == '-') {
+ negated = 1;
+ string++;
+ len--;
+ if (!len)
+ warning(_("invalid value for variable %s"), var);
+ }
+
+ if (!len)
+ break;
+
+ for (i = 0; i < ARRAY_SIZE(fsync_component_table); ++i) {
+ const struct fsync_component_entry *entry = &fsync_component_table[i];
+
+ if (strncmp(entry->name, string, len))
+ continue;
+
+ found = 1;
+ if (negated)
+ output &= ~entry->component_bits;
+ else
+ output |= entry->component_bits;
+ }
+
+ if (!found) {
+ char *component = xstrndup(string, len);
+ warning(_("unknown %s value '%s'"), var, component);
+ free(component);
+ }
+
+ string = ep;
+ }
+
+ return output;
+}
+
int git_parse_maybe_bool(const char *value)
{
int v = git_parse_maybe_bool_text(value);
@@ -1490,6 +1557,13 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
return 0;
}
+ if (!strcmp(var, "core.fsync")) {
+ if (!value)
+ return config_error_nonbool(var);
+ fsync_components = parse_fsync_components(var, value);
+ return 0;
+ }
+
if (!strcmp(var, "core.fsyncmethod")) {
if (!value)
return config_error_nonbool(var);
@@ -1503,7 +1577,7 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
}
if (!strcmp(var, "core.fsyncobjectfiles")) {
- fsync_object_files = git_config_bool(var, value);
+ warning(_("core.fsyncobjectfiles is deprecated; use core.fsync instead"));
return 0;
}
diff --git a/csum-file.c b/csum-file.c
index 26e8a6df44e..59ef3398ca2 100644
--- a/csum-file.c
+++ b/csum-file.c
@@ -58,7 +58,8 @@ static void free_hashfile(struct hashfile *f)
free(f);
}
-int finalize_hashfile(struct hashfile *f, unsigned char *result, unsigned int flags)
+int finalize_hashfile(struct hashfile *f, unsigned char *result,
+ enum fsync_component component, unsigned int flags)
{
int fd;
@@ -69,7 +70,7 @@ int finalize_hashfile(struct hashfile *f, unsigned char *result, unsigned int fl
if (flags & CSUM_HASH_IN_STREAM)
flush(f, f->buffer, the_hash_algo->rawsz);
if (flags & CSUM_FSYNC)
- fsync_or_die(f->fd, f->name);
+ fsync_component_or_die(component, f->fd, f->name);
if (flags & CSUM_CLOSE) {
if (close(f->fd))
die_errno("%s: sha1 file error on close", f->name);
diff --git a/csum-file.h b/csum-file.h
index 291215b34eb..0d29f528fbc 100644
--- a/csum-file.h
+++ b/csum-file.h
@@ -1,6 +1,7 @@
#ifndef CSUM_FILE_H
#define CSUM_FILE_H
+#include "cache.h"
#include "hash.h"
struct progress;
@@ -38,7 +39,7 @@ int hashfile_truncate(struct hashfile *, struct hashfile_checkpoint *);
struct hashfile *hashfd(int fd, const char *name);
struct hashfile *hashfd_check(const char *name);
struct hashfile *hashfd_throughput(int fd, const char *name, struct progress *tp);
-int finalize_hashfile(struct hashfile *, unsigned char *, unsigned int);
+int finalize_hashfile(struct hashfile *, unsigned char *, enum fsync_component, unsigned int);
void hashwrite(struct hashfile *, const void *, unsigned int);
void hashflush(struct hashfile *f);
void crc32_begin(struct hashfile *);
diff --git a/environment.c b/environment.c
index f9140e842cf..09905adecf9 100644
--- a/environment.c
+++ b/environment.c
@@ -42,6 +42,7 @@ const char *git_hooks_path;
int zlib_compression_level = Z_BEST_SPEED;
int pack_compression_level = Z_DEFAULT_COMPRESSION;
enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT;
+enum fsync_component fsync_components = FSYNC_COMPONENTS_DEFAULT;
size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE;
size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT;
size_t delta_base_cache_limit = 96 * 1024 * 1024;
diff --git a/midx.c b/midx.c
index 837b46b2af5..882f91f7d57 100644
--- a/midx.c
+++ b/midx.c
@@ -1406,7 +1406,8 @@ static int write_midx_internal(const char *object_dir,
write_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs);
write_chunkfile(cf, &ctx);
- finalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);
+ finalize_hashfile(f, midx_hash, FSYNC_COMPONENT_PACK_METADATA,
+ CSUM_FSYNC | CSUM_HASH_IN_STREAM);
free_chunkfile(cf);
if (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP))
diff --git a/object-file.c b/object-file.c
index eb972cdccd2..9d9c4a39e85 100644
--- a/object-file.c
+++ b/object-file.c
@@ -1809,8 +1809,7 @@ int hash_object_file(const struct git_hash_algo *algo, const void *buf,
/* Finalize a file on disk, and close it. */
static void close_loose_object(int fd)
{
- if (fsync_object_files)
- fsync_or_die(fd, "loose object file");
+ fsync_component_or_die(FSYNC_COMPONENT_LOOSE_OBJECT, fd, "loose object file");
if (close(fd) != 0)
die_errno(_("error when closing loose object file"));
}
diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c
index 9c55c1531e1..c16e43d1669 100644
--- a/pack-bitmap-write.c
+++ b/pack-bitmap-write.c
@@ -719,7 +719,8 @@ void bitmap_writer_finish(struct pack_idx_entry **index,
if (options & BITMAP_OPT_HASH_CACHE)
write_hash_cache(f, index, index_nr);
- finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
+ finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA,
+ CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
if (adjust_shared_perm(tmp_file.buf))
die_errno("unable to make temporary bitmap file readable");
diff --git a/pack-write.c b/pack-write.c
index a5846f3a346..51812cb1299 100644
--- a/pack-write.c
+++ b/pack-write.c
@@ -159,9 +159,9 @@ const char *write_idx_file(const char *index_name, struct pack_idx_entry **objec
}
hashwrite(f, sha1, the_hash_algo->rawsz);
- finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE |
- ((opts->flags & WRITE_IDX_VERIFY)
- ? 0 : CSUM_FSYNC));
+ finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA,
+ CSUM_HASH_IN_STREAM | CSUM_CLOSE |
+ ((opts->flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC));
return index_name;
}
@@ -281,8 +281,9 @@ const char *write_rev_file_order(const char *rev_name,
if (rev_name && adjust_shared_perm(rev_name) < 0)
die(_("failed to make %s readable"), rev_name);
- finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE |
- ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC));
+ finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA,
+ CSUM_HASH_IN_STREAM | CSUM_CLOSE |
+ ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC));
return rev_name;
}
@@ -390,7 +391,7 @@ void fixup_pack_header_footer(int pack_fd,
the_hash_algo->final_fn(partial_pack_hash, &old_hash_ctx);
the_hash_algo->final_fn(new_pack_hash, &new_hash_ctx);
write_or_die(pack_fd, new_pack_hash, the_hash_algo->rawsz);
- fsync_or_die(pack_fd, pack_name);
+ fsync_component_or_die(FSYNC_COMPONENT_PACK, pack_fd, pack_name);
}
char *index_pack_lockfile(int ip_out, int *is_well_formed)
diff --git a/read-cache.c b/read-cache.c
index f3986596623..f3539681f49 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -3060,7 +3060,7 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile,
return -1;
}
- finalize_hashfile(f, istate->oid.hash, CSUM_HASH_IN_STREAM);
+ finalize_hashfile(f, istate->oid.hash, FSYNC_COMPONENT_NONE, CSUM_HASH_IN_STREAM);
if (close_tempfile_gently(tempfile)) {
error(_("could not close '%s'"), get_tempfile_path(tempfile));
return -1;
--
gitgitgadget
^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v2 3/3] core.fsync: new option to harden the index
2021-12-07 2:46 ` [PATCH v2 0/3] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget
2021-12-07 2:46 ` [PATCH v2 1/3] core.fsyncmethod: add writeout-only mode Neeraj Singh via GitGitGadget
2021-12-07 2:46 ` [PATCH v2 2/3] core.fsync: introduce granular fsync control Neeraj Singh via GitGitGadget
@ 2021-12-07 2:46 ` Neeraj Singh via GitGitGadget
2021-12-07 11:56 ` [PATCH v2 0/3] A design for future-proofing fsync() configuration Patrick Steinhardt
2021-12-09 0:57 ` [PATCH v3 0/4] " Neeraj K. Singh via GitGitGadget
4 siblings, 0 replies; 122+ messages in thread
From: Neeraj Singh via GitGitGadget @ 2021-12-07 2:46 UTC (permalink / raw)
To: git
Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps,
Neeraj K. Singh, Neeraj Singh
From: Neeraj Singh <neerajsi@microsoft.com>
This commit introduces the new ability for the user to harden
the index. In the event of a system crash, the index must be
durable for the user to actually find a file that has been added
to the repo and then deleted from the working tree.
We use the presence of the COMMIT_LOCK flag and absence of the
alternate_index_output as a proxy for determining whether we're
updating the persistent index of the repo or some temporary
index. We don't sync these temporary indexes.
Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
---
Documentation/config/core.txt | 1 +
cache.h | 4 +++-
config.c | 1 +
read-cache.c | 19 +++++++++++++------
4 files changed, 18 insertions(+), 7 deletions(-)
diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
index 4f1747ec871..8e5b7a795ab 100644
--- a/Documentation/config/core.txt
+++ b/Documentation/config/core.txt
@@ -561,6 +561,7 @@ core.fsync::
* `pack` hardens objects added to the repo in packfile form.
* `pack-metadata` hardens packfile bitmaps and indexes.
* `commit-graph` hardens the commit graph file.
+* `index` hardens the index when it is modified.
* `objects` is an aggregate option that includes `loose-objects`, `pack`,
`pack-metadata`, and `commit-graph`.
* `default` is an aggregate option that is equivalent to `objects,-loose-object`
diff --git a/cache.h b/cache.h
index d83fbaf2619..4dc26d7b2c9 100644
--- a/cache.h
+++ b/cache.h
@@ -996,6 +996,7 @@ enum fsync_component {
FSYNC_COMPONENT_PACK = 1 << 1,
FSYNC_COMPONENT_PACK_METADATA = 1 << 2,
FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3,
+ FSYNC_COMPONENT_INDEX = 1 << 4,
};
#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \
@@ -1010,7 +1011,8 @@ enum fsync_component {
#define FSYNC_COMPONENTS_ALL (FSYNC_COMPONENT_LOOSE_OBJECT | \
FSYNC_COMPONENT_PACK | \
FSYNC_COMPONENT_PACK_METADATA | \
- FSYNC_COMPONENT_COMMIT_GRAPH)
+ FSYNC_COMPONENT_COMMIT_GRAPH | \
+ FSYNC_COMPONENT_INDEX)
/*
diff --git a/config.c b/config.c
index 29c867aab03..17039fa9c10 100644
--- a/config.c
+++ b/config.c
@@ -1221,6 +1221,7 @@ static const struct fsync_component_entry {
{ "pack", FSYNC_COMPONENT_PACK },
{ "pack-metadata", FSYNC_COMPONENT_PACK_METADATA },
{ "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH },
+ { "index", FSYNC_COMPONENT_INDEX },
{ "objects", FSYNC_COMPONENTS_OBJECTS },
{ "default", FSYNC_COMPONENTS_DEFAULT },
{ "all", FSYNC_COMPONENTS_ALL },
diff --git a/read-cache.c b/read-cache.c
index f3539681f49..783cb3ea5db 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -2816,7 +2816,7 @@ static int record_ieot(void)
* rely on it.
*/
static int do_write_index(struct index_state *istate, struct tempfile *tempfile,
- int strip_extensions)
+ int strip_extensions, unsigned flags)
{
uint64_t start = getnanotime();
struct hashfile *f;
@@ -2830,6 +2830,7 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile,
struct strbuf previous_name_buf = STRBUF_INIT, *previous_name;
int drop_cache_tree = istate->drop_cache_tree;
off_t offset;
+ int csum_fsync_flag;
int ieot_entries = 1;
struct index_entry_offset_table *ieot = NULL;
int nr, nr_threads;
@@ -3060,7 +3061,13 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile,
return -1;
}
- finalize_hashfile(f, istate->oid.hash, FSYNC_COMPONENT_NONE, CSUM_HASH_IN_STREAM);
+ csum_fsync_flag = 0;
+ if (!alternate_index_output && (flags & COMMIT_LOCK))
+ csum_fsync_flag = CSUM_FSYNC;
+
+ finalize_hashfile(f, istate->oid.hash, FSYNC_COMPONENT_INDEX,
+ CSUM_HASH_IN_STREAM | csum_fsync_flag);
+
if (close_tempfile_gently(tempfile)) {
error(_("could not close '%s'"), get_tempfile_path(tempfile));
return -1;
@@ -3115,7 +3122,7 @@ static int do_write_locked_index(struct index_state *istate, struct lock_file *l
*/
trace2_region_enter_printf("index", "do_write_index", the_repository,
"%s", get_lock_file_path(lock));
- ret = do_write_index(istate, lock->tempfile, 0);
+ ret = do_write_index(istate, lock->tempfile, 0, flags);
trace2_region_leave_printf("index", "do_write_index", the_repository,
"%s", get_lock_file_path(lock));
@@ -3209,7 +3216,7 @@ static int clean_shared_index_files(const char *current_hex)
}
static int write_shared_index(struct index_state *istate,
- struct tempfile **temp)
+ struct tempfile **temp, unsigned flags)
{
struct split_index *si = istate->split_index;
int ret, was_full = !istate->sparse_index;
@@ -3219,7 +3226,7 @@ static int write_shared_index(struct index_state *istate,
trace2_region_enter_printf("index", "shared/do_write_index",
the_repository, "%s", get_tempfile_path(*temp));
- ret = do_write_index(si->base, *temp, 1);
+ ret = do_write_index(si->base, *temp, 1, flags);
trace2_region_leave_printf("index", "shared/do_write_index",
the_repository, "%s", get_tempfile_path(*temp));
@@ -3328,7 +3335,7 @@ int write_locked_index(struct index_state *istate, struct lock_file *lock,
ret = do_write_locked_index(istate, lock, flags);
goto out;
}
- ret = write_shared_index(istate, &temp);
+ ret = write_shared_index(istate, &temp, flags);
saved_errno = errno;
if (is_tempfile_active(temp))
--
gitgitgadget
^ permalink raw reply related [flat|nested] 122+ messages in thread
* Re: [PATCH v2 1/3] core.fsyncmethod: add writeout-only mode
2021-12-07 2:46 ` [PATCH v2 1/3] core.fsyncmethod: add writeout-only mode Neeraj Singh via GitGitGadget
@ 2021-12-07 11:44 ` Patrick Steinhardt
2021-12-07 12:14 ` Ævar Arnfjörð Bjarmason
2021-12-07 23:29 ` Neeraj Singh
2021-12-07 12:18 ` Ævar Arnfjörð Bjarmason
1 sibling, 2 replies; 122+ messages in thread
From: Patrick Steinhardt @ 2021-12-07 11:44 UTC (permalink / raw)
To: Neeraj Singh via GitGitGadget
Cc: git, rsbecker, bagasdotme, newren, avarab, nksingh85,
Neeraj K. Singh
[-- Attachment #1: Type: text/plain, Size: 4326 bytes --]
On Tue, Dec 07, 2021 at 02:46:49AM +0000, Neeraj Singh via GitGitGadget wrote:
> From: Neeraj Singh <neerajsi@microsoft.com>
[snip]
> --- a/compat/mingw.h
> +++ b/compat/mingw.h
> @@ -329,6 +329,9 @@ int mingw_getpagesize(void);
> #define getpagesize mingw_getpagesize
> #endif
>
> +int win32_fsync_no_flush(int fd);
> +#define fsync_no_flush win32_fsync_no_flush
> +
> struct rlimit {
> unsigned int rlim_cur;
> };
> diff --git a/compat/win32/flush.c b/compat/win32/flush.c
> new file mode 100644
> index 00000000000..75324c24ee7
> --- /dev/null
> +++ b/compat/win32/flush.c
> @@ -0,0 +1,28 @@
> +#include "../../git-compat-util.h"
> +#include <winternl.h>
> +#include "lazyload.h"
> +
> +int win32_fsync_no_flush(int fd)
> +{
> + IO_STATUS_BLOCK io_status;
> +
> +#define FLUSH_FLAGS_FILE_DATA_ONLY 1
> +
> + DECLARE_PROC_ADDR(ntdll.dll, NTSTATUS, NtFlushBuffersFileEx,
> + HANDLE FileHandle, ULONG Flags, PVOID Parameters, ULONG ParameterSize,
> + PIO_STATUS_BLOCK IoStatusBlock);
> +
> + if (!INIT_PROC_ADDR(NtFlushBuffersFileEx)) {
> + errno = ENOSYS;
> + return -1;
> + }
I'm wondering whether it would make sense to fall back to fsync(3P) in
case we cannot use writeout-only, but I see that were doing essentially
that in `fsync_or_die()`. There is no indicator to the user though that
writeout-only doesn't work -- do we want to print a one-time warning?
> + memset(&io_status, 0, sizeof(io_status));
> + if (NtFlushBuffersFileEx((HANDLE)_get_osfhandle(fd), FLUSH_FLAGS_FILE_DATA_ONLY,
> + NULL, 0, &io_status)) {
> + errno = EINVAL;
> + return -1;
> + }
> +
> + return 0;
> +}
[snip]
> diff --git a/wrapper.c b/wrapper.c
> index 36e12119d76..1c5f2c87791 100644
> --- a/wrapper.c
> +++ b/wrapper.c
> @@ -546,6 +546,62 @@ int xmkstemp_mode(char *filename_template, int mode)
> return fd;
> }
>
> +int git_fsync(int fd, enum fsync_action action)
> +{
> + switch (action) {
> + case FSYNC_WRITEOUT_ONLY:
> +
> +#ifdef __APPLE__
> + /*
> + * on macOS, fsync just causes filesystem cache writeback but does not
> + * flush hardware caches.
> + */
> + return fsync(fd);
Below we're looping around `EINTR` -- are Apple systems never returning
it?
Patrick
> +#endif
> +
> +#ifdef HAVE_SYNC_FILE_RANGE
> + /*
> + * On linux 2.6.17 and above, sync_file_range is the way to issue
> + * a writeback without a hardware flush. An offset of 0 and size of 0
> + * indicates writeout of the entire file and the wait flags ensure that all
> + * dirty data is written to the disk (potentially in a disk-side cache)
> + * before we continue.
> + */
> +
> + return sync_file_range(fd, 0, 0, SYNC_FILE_RANGE_WAIT_BEFORE |
> + SYNC_FILE_RANGE_WRITE |
> + SYNC_FILE_RANGE_WAIT_AFTER);
> +#endif
> +
> +#ifdef fsync_no_flush
> + return fsync_no_flush(fd);
> +#endif
> +
> + errno = ENOSYS;
> + return -1;
> +
> + case FSYNC_HARDWARE_FLUSH:
> + /*
> + * On some platforms fsync may return EINTR. Try again in this
> + * case, since callers asking for a hardware flush may die if
> + * this function returns an error.
> + */
> + for (;;) {
> + int err;
> +#ifdef __APPLE__
> + err = fcntl(fd, F_FULLFSYNC);
> +#else
> + err = fsync(fd);
> +#endif
> + if (err >= 0 || errno != EINTR)
> + return err;
> + }
> +
> + default:
> + BUG("unexpected git_fsync(%d) call", action);
> + }
> +}
> +
> static int warn_if_unremovable(const char *op, const char *file, int rc)
> {
> int err;
> diff --git a/write-or-die.c b/write-or-die.c
> index 0b1ec8190b6..0702acdd5e8 100644
> --- a/write-or-die.c
> +++ b/write-or-die.c
> @@ -57,10 +57,12 @@ void fprintf_or_die(FILE *f, const char *fmt, ...)
>
> void fsync_or_die(int fd, const char *msg)
> {
> - while (fsync(fd) < 0) {
> - if (errno != EINTR)
> - die_errno("fsync error on '%s'", msg);
> - }
> + if (fsync_method == FSYNC_METHOD_WRITEOUT_ONLY &&
> + git_fsync(fd, FSYNC_WRITEOUT_ONLY) >= 0)
> + return;
> +
> + if (git_fsync(fd, FSYNC_HARDWARE_FLUSH) < 0)
> + die_errno("fsync error on '%s'", msg);
> }
>
> void write_or_die(int fd, const void *buf, size_t count)
> --
> gitgitgadget
>
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v2 2/3] core.fsync: introduce granular fsync control
2021-12-07 2:46 ` [PATCH v2 2/3] core.fsync: introduce granular fsync control Neeraj Singh via GitGitGadget
@ 2021-12-07 11:53 ` Patrick Steinhardt
2021-12-07 20:46 ` Neeraj Singh
2021-12-07 12:29 ` Ævar Arnfjörð Bjarmason
1 sibling, 1 reply; 122+ messages in thread
From: Patrick Steinhardt @ 2021-12-07 11:53 UTC (permalink / raw)
To: Neeraj Singh via GitGitGadget
Cc: git, rsbecker, bagasdotme, newren, avarab, nksingh85,
Neeraj K. Singh
[-- Attachment #1: Type: text/plain, Size: 11137 bytes --]
On Tue, Dec 07, 2021 at 02:46:50AM +0000, Neeraj Singh via GitGitGadget wrote:
> From: Neeraj Singh <neerajsi@microsoft.com>
[snip]
> diff --git a/builtin/index-pack.c b/builtin/index-pack.c
> index c23d01de7dc..c32534c13b4 100644
> --- a/builtin/index-pack.c
> +++ b/builtin/index-pack.c
> @@ -1286,7 +1286,7 @@ static void conclude_pack(int fix_thin_pack, const char *curr_pack, unsigned cha
> nr_objects - nr_objects_initial);
> stop_progress_msg(&progress, msg.buf);
> strbuf_release(&msg);
> - finalize_hashfile(f, tail_hash, 0);
> + finalize_hashfile(f, tail_hash, FSYNC_COMPONENT_PACK, 0);
> hashcpy(read_hash, pack_hash);
> fixup_pack_header_footer(output_fd, pack_hash,
> curr_pack, nr_objects,
> @@ -1508,7 +1508,7 @@ static void final(const char *final_pack_name, const char *curr_pack_name,
> if (!from_stdin) {
> close(input_fd);
> } else {
> - fsync_or_die(output_fd, curr_pack_name);
> + fsync_component_or_die(FSYNC_COMPONENT_PACK, output_fd, curr_pack_name);
> err = close(output_fd);
> if (err)
> die_errno(_("error while closing pack file"));
> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
> index 857be7826f3..916c55d6ce9 100644
> --- a/builtin/pack-objects.c
> +++ b/builtin/pack-objects.c
> @@ -1204,11 +1204,13 @@ static void write_pack_file(void)
> * If so, rewrite it like in fast-import
> */
> if (pack_to_stdout) {
> - finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_CLOSE);
> + finalize_hashfile(f, hash, FSYNC_COMPONENT_NONE,
> + CSUM_HASH_IN_STREAM | CSUM_CLOSE);
It doesn't have any effect here given that we don't sync at all when
writing to stdout, but I wonder whether we should set up the component
correctly regardless of that such that it makes for a less confusing
read.
[snip]
> diff --git a/config.c b/config.c
> index c3410b8a868..29c867aab03 100644
> --- a/config.c
> +++ b/config.c
> @@ -1213,6 +1213,73 @@ static int git_parse_maybe_bool_text(const char *value)
> return -1;
> }
>
> +static const struct fsync_component_entry {
> + const char *name;
> + enum fsync_component component_bits;
> +} fsync_component_table[] = {
> + { "loose-object", FSYNC_COMPONENT_LOOSE_OBJECT },
> + { "pack", FSYNC_COMPONENT_PACK },
> + { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA },
> + { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH },
> + { "objects", FSYNC_COMPONENTS_OBJECTS },
> + { "default", FSYNC_COMPONENTS_DEFAULT },
> + { "all", FSYNC_COMPONENTS_ALL },
> +};
> +
> +static enum fsync_component parse_fsync_components(const char *var, const char *string)
> +{
> + enum fsync_component output = 0;
> +
> + if (!strcmp(string, "none"))
> + return output;
> +
> + while (string) {
> + int i;
> + size_t len;
> + const char *ep;
> + int negated = 0;
> + int found = 0;
> +
> + string = string + strspn(string, ", \t\n\r");
> + ep = strchrnul(string, ',');
> + len = ep - string;
> +
> + if (*string == '-') {
> + negated = 1;
> + string++;
> + len--;
> + if (!len)
> + warning(_("invalid value for variable %s"), var);
> + }
> +
> + if (!len)
> + break;
> +
> + for (i = 0; i < ARRAY_SIZE(fsync_component_table); ++i) {
> + const struct fsync_component_entry *entry = &fsync_component_table[i];
> +
> + if (strncmp(entry->name, string, len))
> + continue;
> +
> + found = 1;
> + if (negated)
> + output &= ~entry->component_bits;
> + else
> + output |= entry->component_bits;
> + }
> +
> + if (!found) {
> + char *component = xstrndup(string, len);
> + warning(_("unknown %s value '%s'"), var, component);
> + free(component);
> + }
> +
> + string = ep;
> + }
> +
> + return output;
> +}
> +
> int git_parse_maybe_bool(const char *value)
> {
> int v = git_parse_maybe_bool_text(value);
> @@ -1490,6 +1557,13 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
> return 0;
> }
>
> + if (!strcmp(var, "core.fsync")) {
> + if (!value)
> + return config_error_nonbool(var);
> + fsync_components = parse_fsync_components(var, value);
> + return 0;
> + }
> +
> if (!strcmp(var, "core.fsyncmethod")) {
> if (!value)
> return config_error_nonbool(var);
> @@ -1503,7 +1577,7 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
> }
>
> if (!strcmp(var, "core.fsyncobjectfiles")) {
> - fsync_object_files = git_config_bool(var, value);
> + warning(_("core.fsyncobjectfiles is deprecated; use core.fsync instead"));
> return 0;
> }
Shouldn't we continue to support this for now such that users can
migrate from the old, deprecated value first before we start to ignore
it?
Patrick
> diff --git a/csum-file.c b/csum-file.c
> index 26e8a6df44e..59ef3398ca2 100644
> --- a/csum-file.c
> +++ b/csum-file.c
> @@ -58,7 +58,8 @@ static void free_hashfile(struct hashfile *f)
> free(f);
> }
>
> -int finalize_hashfile(struct hashfile *f, unsigned char *result, unsigned int flags)
> +int finalize_hashfile(struct hashfile *f, unsigned char *result,
> + enum fsync_component component, unsigned int flags)
> {
> int fd;
>
> @@ -69,7 +70,7 @@ int finalize_hashfile(struct hashfile *f, unsigned char *result, unsigned int fl
> if (flags & CSUM_HASH_IN_STREAM)
> flush(f, f->buffer, the_hash_algo->rawsz);
> if (flags & CSUM_FSYNC)
> - fsync_or_die(f->fd, f->name);
> + fsync_component_or_die(component, f->fd, f->name);
> if (flags & CSUM_CLOSE) {
> if (close(f->fd))
> die_errno("%s: sha1 file error on close", f->name);
> diff --git a/csum-file.h b/csum-file.h
> index 291215b34eb..0d29f528fbc 100644
> --- a/csum-file.h
> +++ b/csum-file.h
> @@ -1,6 +1,7 @@
> #ifndef CSUM_FILE_H
> #define CSUM_FILE_H
>
> +#include "cache.h"
> #include "hash.h"
>
> struct progress;
> @@ -38,7 +39,7 @@ int hashfile_truncate(struct hashfile *, struct hashfile_checkpoint *);
> struct hashfile *hashfd(int fd, const char *name);
> struct hashfile *hashfd_check(const char *name);
> struct hashfile *hashfd_throughput(int fd, const char *name, struct progress *tp);
> -int finalize_hashfile(struct hashfile *, unsigned char *, unsigned int);
> +int finalize_hashfile(struct hashfile *, unsigned char *, enum fsync_component, unsigned int);
> void hashwrite(struct hashfile *, const void *, unsigned int);
> void hashflush(struct hashfile *f);
> void crc32_begin(struct hashfile *);
> diff --git a/environment.c b/environment.c
> index f9140e842cf..09905adecf9 100644
> --- a/environment.c
> +++ b/environment.c
> @@ -42,6 +42,7 @@ const char *git_hooks_path;
> int zlib_compression_level = Z_BEST_SPEED;
> int pack_compression_level = Z_DEFAULT_COMPRESSION;
> enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT;
> +enum fsync_component fsync_components = FSYNC_COMPONENTS_DEFAULT;
> size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE;
> size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT;
> size_t delta_base_cache_limit = 96 * 1024 * 1024;
> diff --git a/midx.c b/midx.c
> index 837b46b2af5..882f91f7d57 100644
> --- a/midx.c
> +++ b/midx.c
> @@ -1406,7 +1406,8 @@ static int write_midx_internal(const char *object_dir,
> write_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs);
> write_chunkfile(cf, &ctx);
>
> - finalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);
> + finalize_hashfile(f, midx_hash, FSYNC_COMPONENT_PACK_METADATA,
> + CSUM_FSYNC | CSUM_HASH_IN_STREAM);
> free_chunkfile(cf);
>
> if (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP))
> diff --git a/object-file.c b/object-file.c
> index eb972cdccd2..9d9c4a39e85 100644
> --- a/object-file.c
> +++ b/object-file.c
> @@ -1809,8 +1809,7 @@ int hash_object_file(const struct git_hash_algo *algo, const void *buf,
> /* Finalize a file on disk, and close it. */
> static void close_loose_object(int fd)
> {
> - if (fsync_object_files)
> - fsync_or_die(fd, "loose object file");
> + fsync_component_or_die(FSYNC_COMPONENT_LOOSE_OBJECT, fd, "loose object file");
> if (close(fd) != 0)
> die_errno(_("error when closing loose object file"));
> }
> diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c
> index 9c55c1531e1..c16e43d1669 100644
> --- a/pack-bitmap-write.c
> +++ b/pack-bitmap-write.c
> @@ -719,7 +719,8 @@ void bitmap_writer_finish(struct pack_idx_entry **index,
> if (options & BITMAP_OPT_HASH_CACHE)
> write_hash_cache(f, index, index_nr);
>
> - finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
> + finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA,
> + CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
>
> if (adjust_shared_perm(tmp_file.buf))
> die_errno("unable to make temporary bitmap file readable");
> diff --git a/pack-write.c b/pack-write.c
> index a5846f3a346..51812cb1299 100644
> --- a/pack-write.c
> +++ b/pack-write.c
> @@ -159,9 +159,9 @@ const char *write_idx_file(const char *index_name, struct pack_idx_entry **objec
> }
>
> hashwrite(f, sha1, the_hash_algo->rawsz);
> - finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE |
> - ((opts->flags & WRITE_IDX_VERIFY)
> - ? 0 : CSUM_FSYNC));
> + finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA,
> + CSUM_HASH_IN_STREAM | CSUM_CLOSE |
> + ((opts->flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC));
> return index_name;
> }
>
> @@ -281,8 +281,9 @@ const char *write_rev_file_order(const char *rev_name,
> if (rev_name && adjust_shared_perm(rev_name) < 0)
> die(_("failed to make %s readable"), rev_name);
>
> - finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE |
> - ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC));
> + finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA,
> + CSUM_HASH_IN_STREAM | CSUM_CLOSE |
> + ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC));
>
> return rev_name;
> }
> @@ -390,7 +391,7 @@ void fixup_pack_header_footer(int pack_fd,
> the_hash_algo->final_fn(partial_pack_hash, &old_hash_ctx);
> the_hash_algo->final_fn(new_pack_hash, &new_hash_ctx);
> write_or_die(pack_fd, new_pack_hash, the_hash_algo->rawsz);
> - fsync_or_die(pack_fd, pack_name);
> + fsync_component_or_die(FSYNC_COMPONENT_PACK, pack_fd, pack_name);
> }
>
> char *index_pack_lockfile(int ip_out, int *is_well_formed)
> diff --git a/read-cache.c b/read-cache.c
> index f3986596623..f3539681f49 100644
> --- a/read-cache.c
> +++ b/read-cache.c
> @@ -3060,7 +3060,7 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile,
> return -1;
> }
>
> - finalize_hashfile(f, istate->oid.hash, CSUM_HASH_IN_STREAM);
> + finalize_hashfile(f, istate->oid.hash, FSYNC_COMPONENT_NONE, CSUM_HASH_IN_STREAM);
> if (close_tempfile_gently(tempfile)) {
> error(_("could not close '%s'"), get_tempfile_path(tempfile));
> return -1;
> --
> gitgitgadget
>
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v2 0/3] A design for future-proofing fsync() configuration
2021-12-07 2:46 ` [PATCH v2 0/3] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget
` (2 preceding siblings ...)
2021-12-07 2:46 ` [PATCH v2 3/3] core.fsync: new option to harden the index Neeraj Singh via GitGitGadget
@ 2021-12-07 11:56 ` Patrick Steinhardt
2021-12-08 0:44 ` Neeraj Singh
2021-12-09 0:57 ` [PATCH v3 0/4] " Neeraj K. Singh via GitGitGadget
4 siblings, 1 reply; 122+ messages in thread
From: Patrick Steinhardt @ 2021-12-07 11:56 UTC (permalink / raw)
To: Neeraj K. Singh via GitGitGadget
Cc: git, rsbecker, bagasdotme, newren, avarab, nksingh85,
Neeraj K. Singh
[-- Attachment #1: Type: text/plain, Size: 27138 bytes --]
On Tue, Dec 07, 2021 at 02:46:48AM +0000, Neeraj K. Singh via GitGitGadget wrote:
> This is an implementation of an extensible configuration mechanism for
> fsyncing persistent components of a repo.
>
> The main goals are to separate the "what" to sync from the "how". There are
> now two settings: core.fsync - Control the 'what', including the index.
> core.fsyncMethod - Control the 'how'. Currently we support writeout-only and
> full fsync.
>
> Syncing of refs can be layered on top of core.fsync. And batch mode will be
> layered on core.fsyncMethod.
>
> core.fsyncobjectfiles is removed and will issue a deprecation warning if
> it's seen.
>
> I'd like to get agreement on this direction before submitting batch mode to
> the list. The batch mode series is available to view at
> https://github.com/neerajsi-msft/git/pull/1.
>
> Please see [1], [2], and [3] for discussions that led to this series.
>
> V2 changes:
>
> * Updated the documentation for core.fsyncmethod to be less certain.
> writeout-only probably does not do the right thing on Linux.
> * Split out the core.fsync=index change into its own commit.
> * Rename REPO_COMPONENT to FSYNC_COMPONENT. This is really specific to
> fsyncing, so the name should reflect that.
> * Re-add missing Makefile change for SYNC_FILE_RANGE.
> * Tested writeout-only mode, index syncing, and general config settings.
>
> [1] https://lore.kernel.org/git/211110.86r1bogg27.gmgdl@evledraar.gmail.com/
> [2]
> https://lore.kernel.org/git/dd65718814011eb93ccc4428f9882e0f025224a6.1636029491.git.ps@pks.im/
> [3]
> https://lore.kernel.org/git/pull.1076.git.git.1629856292.gitgitgadget@gmail.com/
While I bail from the question of whether we want to grant as much
configurability to the user as this patch series does, I quite like the
implementation. It feels rather straight-forward and it's easy to see
how to extend it to support syncing of other subsystems like the loose
refs.
Thanks!
Patrick
> Neeraj Singh (3):
> core.fsyncmethod: add writeout-only mode
> core.fsync: introduce granular fsync control
> core.fsync: new option to harden the index
>
> Documentation/config/core.txt | 35 +++++++++---
> Makefile | 6 ++
> builtin/fast-import.c | 2 +-
> builtin/index-pack.c | 4 +-
> builtin/pack-objects.c | 8 ++-
> bulk-checkin.c | 5 +-
> cache.h | 48 +++++++++++++++-
> commit-graph.c | 3 +-
> compat/mingw.h | 3 +
> compat/win32/flush.c | 28 +++++++++
> config.c | 89 ++++++++++++++++++++++++++++-
> config.mak.uname | 3 +
> configure.ac | 8 +++
> contrib/buildsystems/CMakeLists.txt | 3 +-
> csum-file.c | 5 +-
> csum-file.h | 3 +-
> environment.c | 3 +-
> git-compat-util.h | 24 ++++++++
> midx.c | 3 +-
> object-file.c | 3 +-
> pack-bitmap-write.c | 3 +-
> pack-write.c | 13 +++--
> read-cache.c | 19 ++++--
> wrapper.c | 56 ++++++++++++++++++
> write-or-die.c | 10 ++--
> 25 files changed, 344 insertions(+), 43 deletions(-)
> create mode 100644 compat/win32/flush.c
>
>
> base-commit: abe6bb3905392d5eb6b01fa6e54d7e784e0522aa
> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1093%2Fneerajsi-msft%2Fns%2Fcore-fsync-v2
> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1093/neerajsi-msft/ns/core-fsync-v2
> Pull-Request: https://github.com/gitgitgadget/git/pull/1093
>
> Range-diff vs v1:
>
> 1: 527380ddc3f ! 1: e79522cbdd4 fsync: add writeout-only mode for fsyncing repo data
> @@ Metadata
> Author: Neeraj Singh <neerajsi@microsoft.com>
>
> ## Commit message ##
> - fsync: add writeout-only mode for fsyncing repo data
> + core.fsyncmethod: add writeout-only mode
> +
> + This commit introduces the `core.fsyncmethod` configuration
> + knob, which can currently be set to `fsync` or `writeout-only`.
>
> The new writeout-only mode attempts to tell the operating system to
> flush its in-memory page cache to the storage hardware without issuing a
> @@ Documentation/config/core.txt: core.whitespace::
> + using fsync and related primitives.
> ++
> +* `fsync` uses the fsync() system call or platform equivalents.
> -+* `writeout-only` issues requests to send the writes to the storage
> -+ hardware, but does not send any FLUSH CACHE request. If the operating system
> -+ does not support the required interfaces, this falls back to fsync().
> ++* `writeout-only` issues pagecache writeback requests, but depending on the
> ++ filesystem and storage hardware, data added to the repository may not be
> ++ durable in the event of a system crash. This is the default mode on macOS.
> +
> core.fsyncObjectFiles::
> This boolean will enable 'fsync()' when writing object files.
> +
>
> + ## Makefile ##
> +@@ Makefile: all::
> + #
> + # Define HAVE_CLOCK_MONOTONIC if your platform has CLOCK_MONOTONIC.
> + #
> ++# Define HAVE_SYNC_FILE_RANGE if your platform has sync_file_range.
> ++#
> + # Define NEEDS_LIBRT if your platform requires linking with librt (glibc version
> + # before 2.17) for clock_gettime and CLOCK_MONOTONIC.
> + #
> +@@ Makefile: ifdef HAVE_CLOCK_MONOTONIC
> + BASIC_CFLAGS += -DHAVE_CLOCK_MONOTONIC
> + endif
> +
> ++ifdef HAVE_SYNC_FILE_RANGE
> ++ BASIC_CFLAGS += -DHAVE_SYNC_FILE_RANGE
> ++endif
> ++
> + ifdef NEEDS_LIBRT
> + EXTLIBS += -lrt
> + endif
> +
> ## cache.h ##
> @@ cache.h: extern int read_replace_refs;
> extern char *git_replace_ref_base;
> 2: 23311a10142 ! 2: ff80a94bf9a core.fsync: introduce granular fsync control
> @@ Commit message
> knob which can be used to control how components of the
> repository are made durable on disk.
>
> - This setting allows future extensibility of components
> - that could be synced in two ways:
> + This setting allows future extensibility of the list of
> + syncable components:
> * We issue a warning rather than an error for unrecognized
> components, so new configs can be used with old Git versions.
> * We support negation, so users can choose one of the default
> aggregate options and then remove components that they don't
> - want.
> + want. The user would then harden any new components added in
> + a Git version update.
>
> - This also support the common request of doing absolutely no
> + This also supports the common request of doing absolutely no
> fysncing with the `core.fsync=none` value, which is expected
> to make the test suite faster.
>
> - This commit introduces the new ability for the user to harden
> - the index, which is a requirement for being able to actually
> - find a file that has been added to the repo and then deleted
> - from the working tree.
> -
> Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
>
> ## Documentation/config/core.txt ##
> @@ Documentation/config/core.txt: core.whitespace::
> + hardened via the core.fsyncMethod when created or modified. You can
> + disable hardening of any component by prefixing it with a '-'. Later
> + items take precedence over earlier ones in the list. For example,
> -+ `core.fsync=all,-index` means "harden everything except the index".
> -+ Items that are not hardened may be lost in the event of an unclean
> -+ system shutdown.
> ++ `core.fsync=all,-pack-metadata` means "harden everything except pack
> ++ metadata." Items that are not hardened may be lost in the event of an
> ++ unclean system shutdown.
> ++
> +* `none` disables fsync completely. This must be specified alone.
> +* `loose-object` hardens objects added to the repo in loose-object form.
> +* `pack` hardens objects added to the repo in packfile form.
> +* `pack-metadata` hardens packfile bitmaps and indexes.
> +* `commit-graph` hardens the commit graph file.
> -+* `index` hardens the index when it is modified.
> +* `objects` is an aggregate option that includes `loose-objects`, `pack`,
> + `pack-metadata`, and `commit-graph`.
> +* `default` is an aggregate option that is equivalent to `objects,-loose-object`
> @@ Documentation/config/core.txt: core.whitespace::
> A value indicating the strategy Git will use to harden repository data
> using fsync and related primitives.
> @@ Documentation/config/core.txt: core.fsyncMethod::
> - hardware, but does not send any FLUSH CACHE request. If the operating system
> - does not support the required interfaces, this falls back to fsync().
> + filesystem and storage hardware, data added to the repository may not be
> + durable in the event of a system crash. This is the default mode on macOS.
>
> -core.fsyncObjectFiles::
> - This boolean will enable 'fsync()' when writing object files.
> @@ builtin/fast-import.c: static void end_packfile(void)
>
> close_pack_windows(pack_data);
> - finalize_hashfile(pack_file, cur_pack_oid.hash, 0);
> -+ finalize_hashfile(pack_file, cur_pack_oid.hash, REPO_COMPONENT_PACK, 0);
> ++ finalize_hashfile(pack_file, cur_pack_oid.hash, FSYNC_COMPONENT_PACK, 0);
> fixup_pack_header_footer(pack_data->pack_fd, pack_data->hash,
> pack_data->pack_name, object_count,
> cur_pack_oid.hash, pack_size);
> @@ builtin/index-pack.c: static void conclude_pack(int fix_thin_pack, const char *c
> stop_progress_msg(&progress, msg.buf);
> strbuf_release(&msg);
> - finalize_hashfile(f, tail_hash, 0);
> -+ finalize_hashfile(f, tail_hash, REPO_COMPONENT_PACK, 0);
> ++ finalize_hashfile(f, tail_hash, FSYNC_COMPONENT_PACK, 0);
> hashcpy(read_hash, pack_hash);
> fixup_pack_header_footer(output_fd, pack_hash,
> curr_pack, nr_objects,
> @@ builtin/index-pack.c: static void final(const char *final_pack_name, const char
> close(input_fd);
> } else {
> - fsync_or_die(output_fd, curr_pack_name);
> -+ fsync_component_or_die(REPO_COMPONENT_PACK, output_fd, curr_pack_name);
> ++ fsync_component_or_die(FSYNC_COMPONENT_PACK, output_fd, curr_pack_name);
> err = close(output_fd);
> if (err)
> die_errno(_("error while closing pack file"));
> @@ builtin/pack-objects.c: static void write_pack_file(void)
> */
> if (pack_to_stdout) {
> - finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_CLOSE);
> -+ finalize_hashfile(f, hash, REPO_COMPONENT_NONE,
> ++ finalize_hashfile(f, hash, FSYNC_COMPONENT_NONE,
> + CSUM_HASH_IN_STREAM | CSUM_CLOSE);
> } else if (nr_written == nr_remaining) {
> - finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
> -+ finalize_hashfile(f, hash, REPO_COMPONENT_PACK,
> ++ finalize_hashfile(f, hash, FSYNC_COMPONENT_PACK,
> + CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
> } else {
> - int fd = finalize_hashfile(f, hash, 0);
> -+ int fd = finalize_hashfile(f, hash, REPO_COMPONENT_PACK, 0);
> ++ int fd = finalize_hashfile(f, hash, FSYNC_COMPONENT_PACK, 0);
> fixup_pack_header_footer(fd, hash, pack_tmp_name,
> nr_written, hash, offset);
> close(fd);
> @@ bulk-checkin.c: static void finish_bulk_checkin(struct bulk_checkin_state *state
> goto clear_exit;
> } else if (state->nr_written == 1) {
> - finalize_hashfile(state->f, hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
> -+ finalize_hashfile(state->f, hash, REPO_COMPONENT_PACK,
> ++ finalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK,
> + CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
> } else {
> - int fd = finalize_hashfile(state->f, hash, 0);
> -+ int fd = finalize_hashfile(state->f, hash, REPO_COMPONENT_PACK, 0);
> ++ int fd = finalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK, 0);
> fixup_pack_header_footer(fd, hash, state->pack_tmp_name,
> state->nr_written, hash,
> state->offset);
> @@ cache.h: void reset_shared_repository(void);
> -extern int fsync_object_files;
> +/*
> + * These values are used to help identify parts of a repository to fsync.
> -+ * REPO_COMPONENT_NONE identifies data that will not be a persistent part of the
> ++ * FSYNC_COMPONENT_NONE identifies data that will not be a persistent part of the
> + * repository and so shouldn't be fsynced.
> + */
> -+enum repo_component {
> -+ REPO_COMPONENT_NONE = 0,
> -+ REPO_COMPONENT_LOOSE_OBJECT = 1 << 0,
> -+ REPO_COMPONENT_PACK = 1 << 1,
> -+ REPO_COMPONENT_PACK_METADATA = 1 << 2,
> -+ REPO_COMPONENT_COMMIT_GRAPH = 1 << 3,
> -+ REPO_COMPONENT_INDEX = 1 << 4,
> ++enum fsync_component {
> ++ FSYNC_COMPONENT_NONE = 0,
> ++ FSYNC_COMPONENT_LOOSE_OBJECT = 1 << 0,
> ++ FSYNC_COMPONENT_PACK = 1 << 1,
> ++ FSYNC_COMPONENT_PACK_METADATA = 1 << 2,
> ++ FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3,
> +};
> +
> -+#define FSYNC_COMPONENTS_DEFAULT (REPO_COMPONENT_PACK | \
> -+ REPO_COMPONENT_PACK_METADATA | \
> -+ REPO_COMPONENT_COMMIT_GRAPH)
> ++#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \
> ++ FSYNC_COMPONENT_PACK_METADATA | \
> ++ FSYNC_COMPONENT_COMMIT_GRAPH)
> +
> -+#define FSYNC_COMPONENTS_OBJECTS (REPO_COMPONENT_LOOSE_OBJECT | \
> -+ REPO_COMPONENT_PACK | \
> -+ REPO_COMPONENT_PACK_METADATA | \
> -+ REPO_COMPONENT_COMMIT_GRAPH)
> ++#define FSYNC_COMPONENTS_OBJECTS (FSYNC_COMPONENT_LOOSE_OBJECT | \
> ++ FSYNC_COMPONENT_PACK | \
> ++ FSYNC_COMPONENT_PACK_METADATA | \
> ++ FSYNC_COMPONENT_COMMIT_GRAPH)
> +
> -+#define FSYNC_COMPONENTS_ALL (REPO_COMPONENT_LOOSE_OBJECT | \
> -+ REPO_COMPONENT_PACK | \
> -+ REPO_COMPONENT_PACK_METADATA | \
> -+ REPO_COMPONENT_COMMIT_GRAPH | \
> -+ REPO_COMPONENT_INDEX)
> ++#define FSYNC_COMPONENTS_ALL (FSYNC_COMPONENT_LOOSE_OBJECT | \
> ++ FSYNC_COMPONENT_PACK | \
> ++ FSYNC_COMPONENT_PACK_METADATA | \
> ++ FSYNC_COMPONENT_COMMIT_GRAPH)
> +
> +
> +/*
> + * A bitmask indicating which components of the repo should be fsynced.
> + */
> -+extern enum repo_component fsync_components;
> ++extern enum fsync_component fsync_components;
>
> enum fsync_method {
> FSYNC_METHOD_FSYNC,
> @@ cache.h: int copy_file_with_time(const char *dst, const char *src, int mode);
> void write_or_die(int fd, const void *buf, size_t count);
> void fsync_or_die(int fd, const char *);
>
> -+inline void fsync_component_or_die(enum repo_component component, int fd, const char *msg)
> ++inline void fsync_component_or_die(enum fsync_component component, int fd, const char *msg)
> +{
> + if (fsync_components & component)
> + fsync_or_die(fd, msg);
> @@ commit-graph.c: static int write_commit_graph_file(struct write_commit_graph_con
>
> close_commit_graph(ctx->r->objects);
> - finalize_hashfile(f, file_hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC);
> -+ finalize_hashfile(f, file_hash, REPO_COMPONENT_COMMIT_GRAPH,
> ++ finalize_hashfile(f, file_hash, FSYNC_COMPONENT_COMMIT_GRAPH,
> + CSUM_HASH_IN_STREAM | CSUM_FSYNC);
> free_chunkfile(cf);
>
> @@ config.c: static int git_parse_maybe_bool_text(const char *value)
>
> +static const struct fsync_component_entry {
> + const char *name;
> -+ enum repo_component component_bits;
> ++ enum fsync_component component_bits;
> +} fsync_component_table[] = {
> -+ { "loose-object", REPO_COMPONENT_LOOSE_OBJECT },
> -+ { "pack", REPO_COMPONENT_PACK },
> -+ { "pack-metadata", REPO_COMPONENT_PACK_METADATA },
> -+ { "commit-graph", REPO_COMPONENT_COMMIT_GRAPH },
> -+ { "index", REPO_COMPONENT_INDEX },
> ++ { "loose-object", FSYNC_COMPONENT_LOOSE_OBJECT },
> ++ { "pack", FSYNC_COMPONENT_PACK },
> ++ { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA },
> ++ { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH },
> + { "objects", FSYNC_COMPONENTS_OBJECTS },
> + { "default", FSYNC_COMPONENTS_DEFAULT },
> + { "all", FSYNC_COMPONENTS_ALL },
> +};
> +
> -+static enum repo_component parse_fsync_components(const char *var, const char *string)
> ++static enum fsync_component parse_fsync_components(const char *var, const char *string)
> +{
> -+ enum repo_component output = 0;
> ++ enum fsync_component output = 0;
> +
> + if (!strcmp(string, "none"))
> + return output;
> @@ csum-file.c: static void free_hashfile(struct hashfile *f)
>
> -int finalize_hashfile(struct hashfile *f, unsigned char *result, unsigned int flags)
> +int finalize_hashfile(struct hashfile *f, unsigned char *result,
> -+ enum repo_component component, unsigned int flags)
> ++ enum fsync_component component, unsigned int flags)
> {
> int fd;
>
> @@ csum-file.c: int finalize_hashfile(struct hashfile *f, unsigned char *result, un
> die_errno("%s: sha1 file error on close", f->name);
>
> ## csum-file.h ##
> +@@
> + #ifndef CSUM_FILE_H
> + #define CSUM_FILE_H
> +
> ++#include "cache.h"
> + #include "hash.h"
> +
> + struct progress;
> @@ csum-file.h: int hashfile_truncate(struct hashfile *, struct hashfile_checkpoint *);
> struct hashfile *hashfd(int fd, const char *name);
> struct hashfile *hashfd_check(const char *name);
> struct hashfile *hashfd_throughput(int fd, const char *name, struct progress *tp);
> -int finalize_hashfile(struct hashfile *, unsigned char *, unsigned int);
> -+int finalize_hashfile(struct hashfile *, unsigned char *, enum repo_component, unsigned int);
> ++int finalize_hashfile(struct hashfile *, unsigned char *, enum fsync_component, unsigned int);
> void hashwrite(struct hashfile *, const void *, unsigned int);
> void hashflush(struct hashfile *f);
> void crc32_begin(struct hashfile *);
> @@ environment.c: const char *git_hooks_path;
> int zlib_compression_level = Z_BEST_SPEED;
> int pack_compression_level = Z_DEFAULT_COMPRESSION;
> enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT;
> -+enum repo_component fsync_components = FSYNC_COMPONENTS_DEFAULT;
> ++enum fsync_component fsync_components = FSYNC_COMPONENTS_DEFAULT;
> size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE;
> size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT;
> size_t delta_base_cache_limit = 96 * 1024 * 1024;
> @@ midx.c: static int write_midx_internal(const char *object_dir,
> write_chunkfile(cf, &ctx);
>
> - finalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);
> -+ finalize_hashfile(f, midx_hash, REPO_COMPONENT_PACK_METADATA,
> ++ finalize_hashfile(f, midx_hash, FSYNC_COMPONENT_PACK_METADATA,
> + CSUM_FSYNC | CSUM_HASH_IN_STREAM);
> free_chunkfile(cf);
>
> @@ object-file.c: int hash_object_file(const struct git_hash_algo *algo, const void
> {
> - if (fsync_object_files)
> - fsync_or_die(fd, "loose object file");
> -+ fsync_component_or_die(REPO_COMPONENT_LOOSE_OBJECT, fd, "loose object file");
> ++ fsync_component_or_die(FSYNC_COMPONENT_LOOSE_OBJECT, fd, "loose object file");
> if (close(fd) != 0)
> die_errno(_("error when closing loose object file"));
> }
> @@ pack-bitmap-write.c: void bitmap_writer_finish(struct pack_idx_entry **index,
> write_hash_cache(f, index, index_nr);
>
> - finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
> -+ finalize_hashfile(f, NULL, REPO_COMPONENT_PACK_METADATA,
> ++ finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA,
> + CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
>
> if (adjust_shared_perm(tmp_file.buf))
> @@ pack-write.c: const char *write_idx_file(const char *index_name, struct pack_idx
> hashwrite(f, sha1, the_hash_algo->rawsz);
> - finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE |
> - ((opts->flags & WRITE_IDX_VERIFY)
> -+ finalize_hashfile(f, NULL, REPO_COMPONENT_PACK_METADATA,
> +- ? 0 : CSUM_FSYNC));
> ++ finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA,
> + CSUM_HASH_IN_STREAM | CSUM_CLOSE |
> -+ ((opts->flags & WRITE_IDX_VERIFY)
> - ? 0 : CSUM_FSYNC));
> ++ ((opts->flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC));
> return index_name;
> }
> +
> @@ pack-write.c: const char *write_rev_file_order(const char *rev_name,
> if (rev_name && adjust_shared_perm(rev_name) < 0)
> die(_("failed to make %s readable"), rev_name);
>
> - finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE |
> - ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC));
> -+ finalize_hashfile(f, NULL, REPO_COMPONENT_PACK_METADATA,
> ++ finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA,
> + CSUM_HASH_IN_STREAM | CSUM_CLOSE |
> + ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC));
>
> @@ pack-write.c: void fixup_pack_header_footer(int pack_fd,
> the_hash_algo->final_fn(new_pack_hash, &new_hash_ctx);
> write_or_die(pack_fd, new_pack_hash, the_hash_algo->rawsz);
> - fsync_or_die(pack_fd, pack_name);
> -+ fsync_component_or_die(REPO_COMPONENT_PACK, pack_fd, pack_name);
> ++ fsync_component_or_die(FSYNC_COMPONENT_PACK, pack_fd, pack_name);
> }
>
> char *index_pack_lockfile(int ip_out, int *is_well_formed)
>
> ## read-cache.c ##
> -@@ read-cache.c: static int record_ieot(void)
> - * rely on it.
> - */
> - static int do_write_index(struct index_state *istate, struct tempfile *tempfile,
> -- int strip_extensions)
> -+ int strip_extensions, unsigned flags)
> - {
> - uint64_t start = getnanotime();
> - struct hashfile *f;
> -@@ read-cache.c: static int do_write_index(struct index_state *istate, struct tempfile *tempfile,
> - struct strbuf previous_name_buf = STRBUF_INIT, *previous_name;
> - int drop_cache_tree = istate->drop_cache_tree;
> - off_t offset;
> -+ int csum_fsync_flag;
> - int ieot_entries = 1;
> - struct index_entry_offset_table *ieot = NULL;
> - int nr, nr_threads;
> @@ read-cache.c: static int do_write_index(struct index_state *istate, struct tempfile *tempfile,
> return -1;
> }
>
> - finalize_hashfile(f, istate->oid.hash, CSUM_HASH_IN_STREAM);
> -+ csum_fsync_flag = 0;
> -+ if (!alternate_index_output && (flags & COMMIT_LOCK))
> -+ csum_fsync_flag = CSUM_FSYNC;
> -+
> -+ finalize_hashfile(f, istate->oid.hash, REPO_COMPONENT_INDEX,
> -+ CSUM_HASH_IN_STREAM | csum_fsync_flag);
> -+
> ++ finalize_hashfile(f, istate->oid.hash, FSYNC_COMPONENT_NONE, CSUM_HASH_IN_STREAM);
> if (close_tempfile_gently(tempfile)) {
> error(_("could not close '%s'"), get_tempfile_path(tempfile));
> return -1;
> -@@ read-cache.c: static int do_write_locked_index(struct index_state *istate, struct lock_file *l
> - */
> - trace2_region_enter_printf("index", "do_write_index", the_repository,
> - "%s", get_lock_file_path(lock));
> -- ret = do_write_index(istate, lock->tempfile, 0);
> -+ ret = do_write_index(istate, lock->tempfile, 0, flags);
> - trace2_region_leave_printf("index", "do_write_index", the_repository,
> - "%s", get_lock_file_path(lock));
> -
> -@@ read-cache.c: static int clean_shared_index_files(const char *current_hex)
> - }
> -
> - static int write_shared_index(struct index_state *istate,
> -- struct tempfile **temp)
> -+ struct tempfile **temp, unsigned flags)
> - {
> - struct split_index *si = istate->split_index;
> - int ret, was_full = !istate->sparse_index;
> -@@ read-cache.c: static int write_shared_index(struct index_state *istate,
> -
> - trace2_region_enter_printf("index", "shared/do_write_index",
> - the_repository, "%s", get_tempfile_path(*temp));
> -- ret = do_write_index(si->base, *temp, 1);
> -+ ret = do_write_index(si->base, *temp, 1, flags);
> - trace2_region_leave_printf("index", "shared/do_write_index",
> - the_repository, "%s", get_tempfile_path(*temp));
> -
> -@@ read-cache.c: int write_locked_index(struct index_state *istate, struct lock_file *lock,
> - ret = do_write_locked_index(istate, lock, flags);
> - goto out;
> - }
> -- ret = write_shared_index(istate, &temp);
> -+ ret = write_shared_index(istate, &temp, flags);
> -
> - saved_errno = errno;
> - if (is_tempfile_active(temp))
> -: ----------- > 3: 86e39b8f8d1 core.fsync: new option to harden the index
>
> --
> gitgitgadget
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v2 1/3] core.fsyncmethod: add writeout-only mode
2021-12-07 11:44 ` Patrick Steinhardt
@ 2021-12-07 12:14 ` Ævar Arnfjörð Bjarmason
2021-12-07 23:29 ` Neeraj Singh
1 sibling, 0 replies; 122+ messages in thread
From: Ævar Arnfjörð Bjarmason @ 2021-12-07 12:14 UTC (permalink / raw)
To: Patrick Steinhardt
Cc: Neeraj Singh via GitGitGadget, git, rsbecker, bagasdotme, newren,
nksingh85, Neeraj K. Singh
On Tue, Dec 07 2021, Patrick Steinhardt wrote:
> [[PGP Signed Part:Undecided]]
> On Tue, Dec 07, 2021 at 02:46:49AM +0000, Neeraj Singh via GitGitGadget wrote:
>> From: Neeraj Singh <neerajsi@microsoft.com>
> [...]
> [snip]
>> diff --git a/wrapper.c b/wrapper.c
>> index 36e12119d76..1c5f2c87791 100644
>> --- a/wrapper.c
>> +++ b/wrapper.c
>> @@ -546,6 +546,62 @@ int xmkstemp_mode(char *filename_template, int mode)
>> return fd;
>> }
>>
>> +int git_fsync(int fd, enum fsync_action action)
>> +{
>> + switch (action) {
>> + case FSYNC_WRITEOUT_ONLY:
>> +
>> +#ifdef __APPLE__
>> + /*
>> + * on macOS, fsync just causes filesystem cache writeback but does not
>> + * flush hardware caches.
>> + */
>> + return fsync(fd);
>
> Below we're looping around `EINTR` -- are Apple systems never returning
> it?
I think so per cccdfd22436 (fsync(): be prepared to see EINTR,
2021-06-04), but I'm not sure, but in any case it would make sense for
this to just call the same loop we've been calling elsewhere. It doesn't
seem to hurt, so we can just do that part in the portable portion of the
code.
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v2 1/3] core.fsyncmethod: add writeout-only mode
2021-12-07 2:46 ` [PATCH v2 1/3] core.fsyncmethod: add writeout-only mode Neeraj Singh via GitGitGadget
2021-12-07 11:44 ` Patrick Steinhardt
@ 2021-12-07 12:18 ` Ævar Arnfjörð Bjarmason
2021-12-07 23:58 ` Neeraj Singh
1 sibling, 1 reply; 122+ messages in thread
From: Ævar Arnfjörð Bjarmason @ 2021-12-07 12:18 UTC (permalink / raw)
To: Neeraj Singh via GitGitGadget
Cc: git, rsbecker, bagasdotme, newren, nksingh85, ps, Neeraj Singh
On Tue, Dec 07 2021, Neeraj Singh via GitGitGadget wrote:
> From: Neeraj Singh <neerajsi@microsoft.com>
>
> This commit introduces the `core.fsyncmethod` configuration
Just a commit msg nit: core.fsyncMethod (I see the docs etc. are using
it camelCased, good..
> diff --git a/compat/win32/flush.c b/compat/win32/flush.c
> new file mode 100644
> index 00000000000..75324c24ee7
> --- /dev/null
> +++ b/compat/win32/flush.c
> @@ -0,0 +1,28 @@
> +#include "../../git-compat-util.h"
nit: Just FWIW I think the better thing is '#include
"git-compat-util.h"', i.e. we're compiling at the top-level and have
added it to -I.
(I know a lot of compat/ and contrib/ and even main-tree stuff does
that, but just FWIW it's not needed).
> + if (!strcmp(var, "core.fsyncmethod")) {
> + if (!value)
> + return config_error_nonbool(var);
> + if (!strcmp(value, "fsync"))
> + fsync_method = FSYNC_METHOD_FSYNC;
> + else if (!strcmp(value, "writeout-only"))
> + fsync_method = FSYNC_METHOD_WRITEOUT_ONLY;
> + else
As a non-nit comment I think this config schema looks great so far.
> + warning(_("unknown %s value '%s'"), var, value);
Just a suggestion maybe something slightly scarier like:
"unknown core.fsyncMethod value '%s'; config from future git version? ignoring requested fsync strategy"
Also using the nicer camelCased version instead of "var" (also helps
translators with context...)
> +int git_fsync(int fd, enum fsync_action action)
> +{
> + switch (action) {
> + case FSYNC_WRITEOUT_ONLY:
> +
> +#ifdef __APPLE__
> + /*
> + * on macOS, fsync just causes filesystem cache writeback but does not
> + * flush hardware caches.
> + */
> + return fsync(fd);
> +#endif
> +
> +#ifdef HAVE_SYNC_FILE_RANGE
> + /*
> + * On linux 2.6.17 and above, sync_file_range is the way to issue
> + * a writeback without a hardware flush. An offset of 0 and size of 0
> + * indicates writeout of the entire file and the wait flags ensure that all
> + * dirty data is written to the disk (potentially in a disk-side cache)
> + * before we continue.
> + */
> +
> + return sync_file_range(fd, 0, 0, SYNC_FILE_RANGE_WAIT_BEFORE |
> + SYNC_FILE_RANGE_WRITE |
> + SYNC_FILE_RANGE_WAIT_AFTER);
> +#endif
> +
> +#ifdef fsync_no_flush
> + return fsync_no_flush(fd);
> +#endif
> +
> + errno = ENOSYS;
> + return -1;
> +
> + case FSYNC_HARDWARE_FLUSH:
> + /*
> + * On some platforms fsync may return EINTR. Try again in this
> + * case, since callers asking for a hardware flush may die if
> + * this function returns an error.
> + */
> + for (;;) {
> + int err;
> +#ifdef __APPLE__
> + err = fcntl(fd, F_FULLFSYNC);
> +#else
> + err = fsync(fd);
> +#endif
> + if (err >= 0 || errno != EINTR)
> + return err;
> + }
> +
> + default:
> + BUG("unexpected git_fsync(%d) call", action);
Don't include such "default" cases, you have an exhaustive "enum", if
you skip it the compiler will check this for you.
> + }
> +}
> +
> static int warn_if_unremovable(const char *op, const char *file, int rc)
Just a code nit: I think it's very much preferred if possible to have as
much of code like this compile on all platforms. See the series at
4002e87cb25 (grep: remove #ifdef NO_PTHREADS, 2018-11-03) is part of for
a good example.
Maybe not worth it in this case since they're not nested ifdef's.
I'm basically thinking of something (also re Patrick's comment on the
2nd patch) where we have a platform_fsync() whose return
value/arguments/whatever capture this "I want to return now" or "you
should be looping" and takes the enum_fsync_action" strategy.
Then the git_fsync() would be the platform-independent looping etc., and
another funciton would do the "one fsync at a time, maybe call me
again".
Maybe it would suck more, just food for thought... :)
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v2 2/3] core.fsync: introduce granular fsync control
2021-12-07 2:46 ` [PATCH v2 2/3] core.fsync: introduce granular fsync control Neeraj Singh via GitGitGadget
2021-12-07 11:53 ` Patrick Steinhardt
@ 2021-12-07 12:29 ` Ævar Arnfjörð Bjarmason
2021-12-07 21:44 ` Neeraj Singh
1 sibling, 1 reply; 122+ messages in thread
From: Ævar Arnfjörð Bjarmason @ 2021-12-07 12:29 UTC (permalink / raw)
To: Neeraj Singh via GitGitGadget
Cc: git, rsbecker, bagasdotme, newren, nksingh85, ps, Neeraj Singh
On Tue, Dec 07 2021, Neeraj Singh via GitGitGadget wrote:
> From: Neeraj Singh <neerajsi@microsoft.com>
>
> This commit introduces the `core.fsync` configuration
> knob which can be used to control how components of the
> repository are made durable on disk.
>
> This setting allows future extensibility of the list of
> syncable components:
> * We issue a warning rather than an error for unrecognized
> components, so new configs can be used with old Git versions.
Looks good!
> * We support negation, so users can choose one of the default
> aggregate options and then remove components that they don't
> want. The user would then harden any new components added in
> a Git version update.
I think this config schema makes sense, but just a (I think important)
comment on the "how" not "what" of it. It's really much better to define
config as:
[some]
key = value
key = value2
Than:
[some]
key = value,value2
The reason is that "git config" has good support for working with
multi-valued stuff, so you can do e.g.:
git config --get-all -z some.key
And you can easily (re)set such config e.g. with --replace-all etc., but
for comma-delimited you (and users) need to do all that work themselves.
Similarly instead of:
some.key = want-this
some.key = -not-this
some.key = but-want-this
I think it's better to just have two lists, one inclusive another
exclusive. E.g. see "log.decorate" and "log.excludeDecoration",
"transfer.hideRefs"
Which would mean:
core.fsync = want-this
core.fsyncExcludes = -not-this
For some value of "fsyncExcludes", maybe "noFsync"? Anyway, just a
suggestion on making this easier for users & the implementation.
> This also supports the common request of doing absolutely no
> fysncing with the `core.fsync=none` value, which is expected
> to make the test suite faster.
Let's just use the git_parse_maybe_bool() or git_parse_maybe_bool_text()
so we'll accept "false", "off", "no" like most other such config?
> Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
> ---
> Documentation/config/core.txt | 27 +++++++++----
> builtin/fast-import.c | 2 +-
> builtin/index-pack.c | 4 +-
> builtin/pack-objects.c | 8 ++--
> bulk-checkin.c | 5 ++-
> cache.h | 39 +++++++++++++++++-
> commit-graph.c | 3 +-
> config.c | 76 ++++++++++++++++++++++++++++++++++-
> csum-file.c | 5 ++-
> csum-file.h | 3 +-
> environment.c | 1 +
> midx.c | 3 +-
> object-file.c | 3 +-
> pack-bitmap-write.c | 3 +-
> pack-write.c | 13 +++---
> read-cache.c | 2 +-
> 16 files changed, 164 insertions(+), 33 deletions(-)
>
> diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
> index dbb134f7136..4f1747ec871 100644
> --- a/Documentation/config/core.txt
> +++ b/Documentation/config/core.txt
> @@ -547,6 +547,25 @@ core.whitespace::
> is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent`
> errors. The default tab width is 8. Allowed values are 1 to 63.
>
> +core.fsync::
> + A comma-separated list of parts of the repository which should be
> + hardened via the core.fsyncMethod when created or modified. You can
> + disable hardening of any component by prefixing it with a '-'. Later
> + items take precedence over earlier ones in the list. For example,
> + `core.fsync=all,-pack-metadata` means "harden everything except pack
> + metadata." Items that are not hardened may be lost in the event of an
> + unclean system shutdown.
> ++
> +* `none` disables fsync completely. This must be specified alone.
> +* `loose-object` hardens objects added to the repo in loose-object form.
> +* `pack` hardens objects added to the repo in packfile form.
> +* `pack-metadata` hardens packfile bitmaps and indexes.
> +* `commit-graph` hardens the commit graph file.
> +* `objects` is an aggregate option that includes `loose-objects`, `pack`,
> + `pack-metadata`, and `commit-graph`.
> +* `default` is an aggregate option that is equivalent to `objects,-loose-object`
> +* `all` is an aggregate option that syncs all individual components above.
> +
It's probably a *bit* more work to set up, but I wonder if this wouldn't
be simpler if we just said (and this is partially going against what I
noted above):
== BEGIN DOC
core.fsync is a multi-value config variable where each item is a
pathspec that'll get matched the same way as 'git-ls-files' et al.
When we sync pretend that a path like .git/objects/de/adbeef... is
relative to the top-level of the git
directory. E.g. "objects/de/adbeaf.." or "objects/pack/...".
You can then supply a list of wildcards and exclusions to configure
syncing. or "false", "off" etc. to turn it off. These are synonymous
with:
; same as "false"
core.fsync = ":!*"
Or:
; same as "true"
core.fsync = "*"
Or, to selectively sync some things and not others:
;; Sync objects, but not "info"
core.fsync = ":!objects/info/**"
core.fsync = "objects/**"
See gitrepository-layout(5) for details about what sort of paths you
might be expected to match. Not all paths listed there will go through
this mechanism (e.g. currently objects do, but nothing to do with config
does).
We can and will match this against "fake paths", e.g. when writing out
packs we may match against just the string "objects/pack", we're not
going to re-check if every packfile we're writing matches your globs,
ditto for loose objects. Be reasonable!
This metharism is intended as a shorthand that provides some flexibility
when fsyncing, while not forcing git to come up with labels for all
paths the git dir, or to support crazyness like "objects/de/adbeef*"
More paths may be added or removed in the future, and we make no
promises that we won't move things around, so if in doubt use
e.g. "true" or a wide pattern match like "objects/**". When in doubt
stick to the golden path of examples provided in this documentation.
== END DOC
It's a tad more complex to set up, but I wonder if that isn't worth
it. It nicely gets around any current and future issues of deciding what
labels such as "loose-object" etc. to pick, as well as slotting into an
existing method of doing exclude/include lists.
> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
> index 857be7826f3..916c55d6ce9 100644
> --- a/builtin/pack-objects.c
> +++ b/builtin/pack-objects.c
> @@ -1204,11 +1204,13 @@ static void write_pack_file(void)
> * If so, rewrite it like in fast-import
> */
> if (pack_to_stdout) {
> - finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_CLOSE);
> + finalize_hashfile(f, hash, FSYNC_COMPONENT_NONE,
> + CSUM_HASH_IN_STREAM | CSUM_CLOSE);
Not really related to this per-se, but since you're touching the API
everything goes through I wonder if callers should just always try to
fsync, and we can just catch EROFS and EINVAL in the wrapper if someone
tries to flush stdout, or catch the fd at that lower level.
Or maybe there's a good reason for this...
> [...]
> +/*
> + * These values are used to help identify parts of a repository to fsync.
> + * FSYNC_COMPONENT_NONE identifies data that will not be a persistent part of the
> + * repository and so shouldn't be fsynced.
> + */
> +enum fsync_component {
> + FSYNC_COMPONENT_NONE = 0,
I haven't read ahead much but in most other such cases we don't define
the "= 0", just start at 1<<0, then check the flags elsewhere...
> +static const struct fsync_component_entry {
> + const char *name;
> + enum fsync_component component_bits;
> +} fsync_component_table[] = {
> + { "loose-object", FSYNC_COMPONENT_LOOSE_OBJECT },
> + { "pack", FSYNC_COMPONENT_PACK },
> + { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA },
> + { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH },
> + { "objects", FSYNC_COMPONENTS_OBJECTS },
> + { "default", FSYNC_COMPONENTS_DEFAULT },
> + { "all", FSYNC_COMPONENTS_ALL },
> +};
> +
> +static enum fsync_component parse_fsync_components(const char *var, const char *string)
> +{
> + enum fsync_component output = 0;
> +
> + if (!strcmp(string, "none"))
> + return output;
> +
> + while (string) {
> + int i;
> + size_t len;
> + const char *ep;
> + int negated = 0;
> + int found = 0;
> +
> + string = string + strspn(string, ", \t\n\r");
Aside from the "use a list" isn't this hardcoding some windows-specific
assumptions with \n\r? Maybe not...
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v2 2/3] core.fsync: introduce granular fsync control
2021-12-07 11:53 ` Patrick Steinhardt
@ 2021-12-07 20:46 ` Neeraj Singh
0 siblings, 0 replies; 122+ messages in thread
From: Neeraj Singh @ 2021-12-07 20:46 UTC (permalink / raw)
To: Patrick Steinhardt
Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker,
Bagas Sanjaya, Elijah Newren,
Ævar Arnfjörð Bjarmason, Neeraj K. Singh
On Tue, Dec 7, 2021 at 3:54 AM Patrick Steinhardt <ps@pks.im> wrote:
>
> On Tue, Dec 07, 2021 at 02:46:50AM +0000, Neeraj Singh via GitGitGadget wrote:
> > From: Neeraj Singh <neerajsi@microsoft.com>
> [snip]
> > diff --git a/builtin/index-pack.c b/builtin/index-pack.c
> > index c23d01de7dc..c32534c13b4 100644
> > --- a/builtin/index-pack.c
> > +++ b/builtin/index-pack.c
> > @@ -1286,7 +1286,7 @@ static void conclude_pack(int fix_thin_pack, const char *curr_pack, unsigned cha
> > nr_objects - nr_objects_initial);
> > stop_progress_msg(&progress, msg.buf);
> > strbuf_release(&msg);
> > - finalize_hashfile(f, tail_hash, 0);
> > + finalize_hashfile(f, tail_hash, FSYNC_COMPONENT_PACK, 0);
> > hashcpy(read_hash, pack_hash);
> > fixup_pack_header_footer(output_fd, pack_hash,
> > curr_pack, nr_objects,
> > @@ -1508,7 +1508,7 @@ static void final(const char *final_pack_name, const char *curr_pack_name,
> > if (!from_stdin) {
> > close(input_fd);
> > } else {
> > - fsync_or_die(output_fd, curr_pack_name);
> > + fsync_component_or_die(FSYNC_COMPONENT_PACK, output_fd, curr_pack_name);
> > err = close(output_fd);
> > if (err)
> > die_errno(_("error while closing pack file"));
> > diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
> > index 857be7826f3..916c55d6ce9 100644
> > --- a/builtin/pack-objects.c
> > +++ b/builtin/pack-objects.c
> > @@ -1204,11 +1204,13 @@ static void write_pack_file(void)
> > * If so, rewrite it like in fast-import
> > */
> > if (pack_to_stdout) {
> > - finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_CLOSE);
> > + finalize_hashfile(f, hash, FSYNC_COMPONENT_NONE,
> > + CSUM_HASH_IN_STREAM | CSUM_CLOSE);
>
> It doesn't have any effect here given that we don't sync at all when
> writing to stdout, but I wonder whether we should set up the component
> correctly regardless of that such that it makes for a less confusing
> read.
>
If it's not actually a file with some name known to git, is it really
a component of the repository? I'd like to leave this one as-is.
> [snip]
> > diff --git a/config.c b/config.c
> > index c3410b8a868..29c867aab03 100644
> > --- a/config.c
> > +++ b/config.c
> > @@ -1213,6 +1213,73 @@ static int git_parse_maybe_bool_text(const char *value)
> > return -1;
> > }
> >
> > +static const struct fsync_component_entry {
> > + const char *name;
> > + enum fsync_component component_bits;
> > +} fsync_component_table[] = {
> > + { "loose-object", FSYNC_COMPONENT_LOOSE_OBJECT },
> > + { "pack", FSYNC_COMPONENT_PACK },
> > + { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA },
> > + { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH },
> > + { "objects", FSYNC_COMPONENTS_OBJECTS },
> > + { "default", FSYNC_COMPONENTS_DEFAULT },
> > + { "all", FSYNC_COMPONENTS_ALL },
> > +};
> > +
> > +static enum fsync_component parse_fsync_components(const char *var, const char *string)
> > +{
> > + enum fsync_component output = 0;
> > +
> > + if (!strcmp(string, "none"))
> > + return output;
> > +
> > + while (string) {
> > + int i;
> > + size_t len;
> > + const char *ep;
> > + int negated = 0;
> > + int found = 0;
> > +
> > + string = string + strspn(string, ", \t\n\r");
> > + ep = strchrnul(string, ',');
> > + len = ep - string;
> > +
> > + if (*string == '-') {
> > + negated = 1;
> > + string++;
> > + len--;
> > + if (!len)
> > + warning(_("invalid value for variable %s"), var);
> > + }
> > +
> > + if (!len)
> > + break;
> > +
> > + for (i = 0; i < ARRAY_SIZE(fsync_component_table); ++i) {
> > + const struct fsync_component_entry *entry = &fsync_component_table[i];
> > +
> > + if (strncmp(entry->name, string, len))
> > + continue;
> > +
> > + found = 1;
> > + if (negated)
> > + output &= ~entry->component_bits;
> > + else
> > + output |= entry->component_bits;
> > + }
> > +
> > + if (!found) {
> > + char *component = xstrndup(string, len);
> > + warning(_("unknown %s value '%s'"), var, component);
> > + free(component);
> > + }
> > +
> > + string = ep;
> > + }
> > +
> > + return output;
> > +}
> > +
> > int git_parse_maybe_bool(const char *value)
> > {
> > int v = git_parse_maybe_bool_text(value);
> > @@ -1490,6 +1557,13 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
> > return 0;
> > }
> >
> > + if (!strcmp(var, "core.fsync")) {
> > + if (!value)
> > + return config_error_nonbool(var);
> > + fsync_components = parse_fsync_components(var, value);
> > + return 0;
> > + }
> > +
> > if (!strcmp(var, "core.fsyncmethod")) {
> > if (!value)
> > return config_error_nonbool(var);
> > @@ -1503,7 +1577,7 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
> > }
> >
> > if (!strcmp(var, "core.fsyncobjectfiles")) {
> > - fsync_object_files = git_config_bool(var, value);
> > + warning(_("core.fsyncobjectfiles is deprecated; use core.fsync instead"));
> > return 0;
> > }
>
> Shouldn't we continue to support this for now such that users can
> migrate from the old, deprecated value first before we start to ignore
> it?
>
> Patrick
>
That's a good question and one I was hoping to answer through this
discussion. I'm guessing that most users do not have this setting
explicitly set, and it's largely a non-functional change. Git only
behaves differently in the extreme corner case of a system crash after
git exits. That's why I believe it's okay to deprecate and remove in
one release.
If we choose to keep supporting the setting, it would introduce a
little complexity in the configuration code, but it should be doable.
I think the right semantics would be to ignore core.fsyncobjectfiles
if core.fsync is specified with any value. Otherwise,
core.fsyncobjectfiles would be equivalent to `default,-loose-object`
or `default,loose-object` for `false` and `true` respectively. I'd
prefer not to do this if I can get away with it :).
Thanks,
Neeraj
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v2 2/3] core.fsync: introduce granular fsync control
2021-12-07 12:29 ` Ævar Arnfjörð Bjarmason
@ 2021-12-07 21:44 ` Neeraj Singh
2021-12-08 10:05 ` Ævar Arnfjörð Bjarmason
0 siblings, 1 reply; 122+ messages in thread
From: Neeraj Singh @ 2021-12-07 21:44 UTC (permalink / raw)
To: Ævar Arnfjörð Bjarmason
Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker,
Bagas Sanjaya, Elijah Newren, Patrick Steinhardt, Neeraj Singh
On Tue, Dec 7, 2021 at 5:01 AM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
>
>
> On Tue, Dec 07 2021, Neeraj Singh via GitGitGadget wrote:
>
> > From: Neeraj Singh <neerajsi@microsoft.com>
> >
> > This commit introduces the `core.fsync` configuration
> > knob which can be used to control how components of the
> > repository are made durable on disk.
> >
> > This setting allows future extensibility of the list of
> > syncable components:
> > * We issue a warning rather than an error for unrecognized
> > components, so new configs can be used with old Git versions.
>
> Looks good!
>
> > * We support negation, so users can choose one of the default
> > aggregate options and then remove components that they don't
> > want. The user would then harden any new components added in
> > a Git version update.
>
> I think this config schema makes sense, but just a (I think important)
> comment on the "how" not "what" of it. It's really much better to define
> config as:
>
> [some]
> key = value
> key = value2
>
> Than:
>
> [some]
> key = value,value2
>
> The reason is that "git config" has good support for working with
> multi-valued stuff, so you can do e.g.:
>
> git config --get-all -z some.key
>
> And you can easily (re)set such config e.g. with --replace-all etc., but
> for comma-delimited you (and users) need to do all that work themselves.
>
> Similarly instead of:
>
> some.key = want-this
> some.key = -not-this
> some.key = but-want-this
>
> I think it's better to just have two lists, one inclusive another
> exclusive. E.g. see "log.decorate" and "log.excludeDecoration",
> "transfer.hideRefs"
>
> Which would mean:
>
> core.fsync = want-this
> core.fsyncExcludes = -not-this
>
> For some value of "fsyncExcludes", maybe "noFsync"? Anyway, just a
> suggestion on making this easier for users & the implementation.
>
Maybe there's some way to handle this I'm unaware of, but a
disadvantage of your multi-valued config proposal is that it's harder,
for example, for a per-repo config store to reasonably override a
per-user config store. With the configuration scheme as-is, I can
have a per-user setting like `core.fsync=all` which covers my typical
repos, but then have a maintainer repo with a private setting of
`core.fsync=none` to speed up cases where I'm mostly working with
other people's changes that are backed up in email or server-side
repos. The latter setting conveniently overrides the former setting
in all aspects.
Also, with the core.fsync and core.fsyncExcludes, how would you spell
"don't sync anything"? Would you still have the aggregate options.?
> > This also supports the common request of doing absolutely no
> > fysncing with the `core.fsync=none` value, which is expected
> > to make the test suite faster.
>
> Let's just use the git_parse_maybe_bool() or git_parse_maybe_bool_text()
> so we'll accept "false", "off", "no" like most other such config?
Junio's previous feedback when discussing batch mode [1] was to offer
less flexibility when parsing new values of these configuration
options. I agree with his statement that "making machine-readable
tokens be spelled in different ways is a 'disease'." I'd like to
leave this as-is so that the documentation can clearly state the exact
set of allowable values.
[1] https://lore.kernel.org/git/xmqqr1dqzyl7.fsf@gitster.g/
>
> > Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
> > ---
> > Documentation/config/core.txt | 27 +++++++++----
> > builtin/fast-import.c | 2 +-
> > builtin/index-pack.c | 4 +-
> > builtin/pack-objects.c | 8 ++--
> > bulk-checkin.c | 5 ++-
> > cache.h | 39 +++++++++++++++++-
> > commit-graph.c | 3 +-
> > config.c | 76 ++++++++++++++++++++++++++++++++++-
> > csum-file.c | 5 ++-
> > csum-file.h | 3 +-
> > environment.c | 1 +
> > midx.c | 3 +-
> > object-file.c | 3 +-
> > pack-bitmap-write.c | 3 +-
> > pack-write.c | 13 +++---
> > read-cache.c | 2 +-
> > 16 files changed, 164 insertions(+), 33 deletions(-)
> >
> > diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
> > index dbb134f7136..4f1747ec871 100644
> > --- a/Documentation/config/core.txt
> > +++ b/Documentation/config/core.txt
> > @@ -547,6 +547,25 @@ core.whitespace::
> > is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent`
> > errors. The default tab width is 8. Allowed values are 1 to 63.
> >
> > +core.fsync::
> > + A comma-separated list of parts of the repository which should be
> > + hardened via the core.fsyncMethod when created or modified. You can
> > + disable hardening of any component by prefixing it with a '-'. Later
> > + items take precedence over earlier ones in the list. For example,
> > + `core.fsync=all,-pack-metadata` means "harden everything except pack
> > + metadata." Items that are not hardened may be lost in the event of an
> > + unclean system shutdown.
> > ++
> > +* `none` disables fsync completely. This must be specified alone.
> > +* `loose-object` hardens objects added to the repo in loose-object form.
> > +* `pack` hardens objects added to the repo in packfile form.
> > +* `pack-metadata` hardens packfile bitmaps and indexes.
> > +* `commit-graph` hardens the commit graph file.
> > +* `objects` is an aggregate option that includes `loose-objects`, `pack`,
> > + `pack-metadata`, and `commit-graph`.
> > +* `default` is an aggregate option that is equivalent to `objects,-loose-object`
> > +* `all` is an aggregate option that syncs all individual components above.
> > +
>
> It's probably a *bit* more work to set up, but I wonder if this wouldn't
> be simpler if we just said (and this is partially going against what I
> noted above):
>
> == BEGIN DOC
>
> core.fsync is a multi-value config variable where each item is a
> pathspec that'll get matched the same way as 'git-ls-files' et al.
>
> When we sync pretend that a path like .git/objects/de/adbeef... is
> relative to the top-level of the git
> directory. E.g. "objects/de/adbeaf.." or "objects/pack/...".
>
> You can then supply a list of wildcards and exclusions to configure
> syncing. or "false", "off" etc. to turn it off. These are synonymous
> with:
>
> ; same as "false"
> core.fsync = ":!*"
>
> Or:
>
> ; same as "true"
> core.fsync = "*"
>
> Or, to selectively sync some things and not others:
>
> ;; Sync objects, but not "info"
> core.fsync = ":!objects/info/**"
> core.fsync = "objects/**"
>
> See gitrepository-layout(5) for details about what sort of paths you
> might be expected to match. Not all paths listed there will go through
> this mechanism (e.g. currently objects do, but nothing to do with config
> does).
>
> We can and will match this against "fake paths", e.g. when writing out
> packs we may match against just the string "objects/pack", we're not
> going to re-check if every packfile we're writing matches your globs,
> ditto for loose objects. Be reasonable!
>
> This metharism is intended as a shorthand that provides some flexibility
> when fsyncing, while not forcing git to come up with labels for all
> paths the git dir, or to support crazyness like "objects/de/adbeef*"
>
> More paths may be added or removed in the future, and we make no
> promises that we won't move things around, so if in doubt use
> e.g. "true" or a wide pattern match like "objects/**". When in doubt
> stick to the golden path of examples provided in this documentation.
>
> == END DOC
>
>
> It's a tad more complex to set up, but I wonder if that isn't worth
> it. It nicely gets around any current and future issues of deciding what
> labels such as "loose-object" etc. to pick, as well as slotting into an
> existing method of doing exclude/include lists.
>
I think this proposal is a lot of complexity to avoid coming up with a
new name for syncable things as they are added to Git. A path based
mechanism makes it hard to document for the (advanced) user what the
full set of things is and how it might change from release to release.
I think the current core.fsync scheme is a bit easier to understand,
query, and extend.
> > diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
> > index 857be7826f3..916c55d6ce9 100644
> > --- a/builtin/pack-objects.c
> > +++ b/builtin/pack-objects.c
> > @@ -1204,11 +1204,13 @@ static void write_pack_file(void)
> > * If so, rewrite it like in fast-import
> > */
> > if (pack_to_stdout) {
> > - finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_CLOSE);
> > + finalize_hashfile(f, hash, FSYNC_COMPONENT_NONE,
> > + CSUM_HASH_IN_STREAM | CSUM_CLOSE);
>
> Not really related to this per-se, but since you're touching the API
> everything goes through I wonder if callers should just always try to
> fsync, and we can just catch EROFS and EINVAL in the wrapper if someone
> tries to flush stdout, or catch the fd at that lower level.
>
> Or maybe there's a good reason for this...
It's platform dependent, but I'd expect fsync would do something for
pipes or stdout redirected to a file. In these cases we really don't
want to fsync since we have no idea what we're talking to and we're
potentially worsening performance for probably no benefit.
> > [...]
> > +/*
> > + * These values are used to help identify parts of a repository to fsync.
> > + * FSYNC_COMPONENT_NONE identifies data that will not be a persistent part of the
> > + * repository and so shouldn't be fsynced.
> > + */
> > +enum fsync_component {
> > + FSYNC_COMPONENT_NONE = 0,
>
> I haven't read ahead much but in most other such cases we don't define
> the "= 0", just start at 1<<0, then check the flags elsewhere...
>
> > +static const struct fsync_component_entry {
> > + const char *name;
> > + enum fsync_component component_bits;
> > +} fsync_component_table[] = {
> > + { "loose-object", FSYNC_COMPONENT_LOOSE_OBJECT },
> > + { "pack", FSYNC_COMPONENT_PACK },
> > + { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA },
> > + { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH },
> > + { "objects", FSYNC_COMPONENTS_OBJECTS },
> > + { "default", FSYNC_COMPONENTS_DEFAULT },
> > + { "all", FSYNC_COMPONENTS_ALL },
> > +};
> > +
> > +static enum fsync_component parse_fsync_components(const char *var, const char *string)
> > +{
> > + enum fsync_component output = 0;
> > +
> > + if (!strcmp(string, "none"))
> > + return output;
> > +
> > + while (string) {
> > + int i;
> > + size_t len;
> > + const char *ep;
> > + int negated = 0;
> > + int found = 0;
> > +
> > + string = string + strspn(string, ", \t\n\r");
>
> Aside from the "use a list" isn't this hardcoding some windows-specific
> assumptions with \n\r? Maybe not...
I shamelessly stole this code from parse_whitespace_rule. I thought
about making a helper to be called by both functions, but the amount
of state going into and out of the wrapper via arguments was
substantial and seemed to negate the benefit of deduplication.
Thanks for the review,
Neeraj
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v2 1/3] core.fsyncmethod: add writeout-only mode
2021-12-07 11:44 ` Patrick Steinhardt
2021-12-07 12:14 ` Ævar Arnfjörð Bjarmason
@ 2021-12-07 23:29 ` Neeraj Singh
1 sibling, 0 replies; 122+ messages in thread
From: Neeraj Singh @ 2021-12-07 23:29 UTC (permalink / raw)
To: Patrick Steinhardt
Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker,
Bagas Sanjaya, Elijah Newren,
Ævar Arnfjörð Bjarmason, Neeraj K. Singh
On Tue, Dec 7, 2021 at 3:45 AM Patrick Steinhardt <ps@pks.im> wrote:
>
> On Tue, Dec 07, 2021 at 02:46:49AM +0000, Neeraj Singh via GitGitGadget wrote:
> > From: Neeraj Singh <neerajsi@microsoft.com>
> [snip]
> > --- a/compat/mingw.h
> > +++ b/compat/mingw.h
> > @@ -329,6 +329,9 @@ int mingw_getpagesize(void);
> > #define getpagesize mingw_getpagesize
> > #endif
> >
> > +int win32_fsync_no_flush(int fd);
> > +#define fsync_no_flush win32_fsync_no_flush
> > +
> > struct rlimit {
> > unsigned int rlim_cur;
> > };
> > diff --git a/compat/win32/flush.c b/compat/win32/flush.c
> > new file mode 100644
> > index 00000000000..75324c24ee7
> > --- /dev/null
> > +++ b/compat/win32/flush.c
> > @@ -0,0 +1,28 @@
> > +#include "../../git-compat-util.h"
> > +#include <winternl.h>
> > +#include "lazyload.h"
> > +
> > +int win32_fsync_no_flush(int fd)
> > +{
> > + IO_STATUS_BLOCK io_status;
> > +
> > +#define FLUSH_FLAGS_FILE_DATA_ONLY 1
> > +
> > + DECLARE_PROC_ADDR(ntdll.dll, NTSTATUS, NtFlushBuffersFileEx,
> > + HANDLE FileHandle, ULONG Flags, PVOID Parameters, ULONG ParameterSize,
> > + PIO_STATUS_BLOCK IoStatusBlock);
> > +
> > + if (!INIT_PROC_ADDR(NtFlushBuffersFileEx)) {
> > + errno = ENOSYS;
> > + return -1;
> > + }
>
> I'm wondering whether it would make sense to fall back to fsync(3P) in
> case we cannot use writeout-only, but I see that were doing essentially
> that in `fsync_or_die()`. There is no indicator to the user though that
> writeout-only doesn't work -- do we want to print a one-time warning?
>
I wanted to leave the fallback to the caller so that the algorithm can
be adjusted in some way based on whether writeout-only succeeded. For
batched fsync object files, we refrain from doing the last fsync if we
were doing real fsyncs all along.
I didn't want to issue a warning, since this writeout-only codepath
might be invoked from multiple subprocesses, which would each
potentially issue their one warning. The consequence of failing
writeout only is worse performance, but should not be compromised
safety, so I'm not sure the user gets enough from the warning to
justify something that's potentially spammy. In practice, when batch
mode is adopted on Windows (by default), some older pre-Win8 systems
will experience fsyncs and equivalent performance to what they're
seeing today. I don't want these users to have a warning too.
> > + memset(&io_status, 0, sizeof(io_status));
> > + if (NtFlushBuffersFileEx((HANDLE)_get_osfhandle(fd), FLUSH_FLAGS_FILE_DATA_ONLY,
> > + NULL, 0, &io_status)) {
> > + errno = EINVAL;
> > + return -1;
> > + }
> > +
> > + return 0;
> > +}
>
> [snip]
> > diff --git a/wrapper.c b/wrapper.c
> > index 36e12119d76..1c5f2c87791 100644
> > --- a/wrapper.c
> > +++ b/wrapper.c
> > @@ -546,6 +546,62 @@ int xmkstemp_mode(char *filename_template, int mode)
> > return fd;
> > }
> >
> > +int git_fsync(int fd, enum fsync_action action)
> > +{
> > + switch (action) {
> > + case FSYNC_WRITEOUT_ONLY:
> > +
> > +#ifdef __APPLE__
> > + /*
> > + * on macOS, fsync just causes filesystem cache writeback but does not
> > + * flush hardware caches.
> > + */
> > + return fsync(fd);
>
> Below we're looping around `EINTR` -- are Apple systems never returning
> it?
>
The EINTR check was added due to a test failure on HP NonStop. I
don't believe any other platform has actually complained about that.
Thanks again for the code review!
-Neeraj
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v2 1/3] core.fsyncmethod: add writeout-only mode
2021-12-07 12:18 ` Ævar Arnfjörð Bjarmason
@ 2021-12-07 23:58 ` Neeraj Singh
0 siblings, 0 replies; 122+ messages in thread
From: Neeraj Singh @ 2021-12-07 23:58 UTC (permalink / raw)
To: Ævar Arnfjörð Bjarmason
Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker,
Bagas Sanjaya, Elijah Newren, Patrick Steinhardt, Neeraj Singh
On Tue, Dec 7, 2021 at 4:27 AM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
>
>
> On Tue, Dec 07 2021, Neeraj Singh via GitGitGadget wrote:
>
> > From: Neeraj Singh <neerajsi@microsoft.com>
> >
> > This commit introduces the `core.fsyncmethod` configuration
>
> Just a commit msg nit: core.fsyncMethod (I see the docs etc. are using
> it camelCased, good..
Will fix.
> > diff --git a/compat/win32/flush.c b/compat/win32/flush.c
> > new file mode 100644
> > index 00000000000..75324c24ee7
> > --- /dev/null
> > +++ b/compat/win32/flush.c
> > @@ -0,0 +1,28 @@
> > +#include "../../git-compat-util.h"
>
> nit: Just FWIW I think the better thing is '#include
> "git-compat-util.h"', i.e. we're compiling at the top-level and have
> added it to -I.
>
> (I know a lot of compat/ and contrib/ and even main-tree stuff does
> that, but just FWIW it's not needed).
>
Will fix.
> > + if (!strcmp(var, "core.fsyncmethod")) {
> > + if (!value)
> > + return config_error_nonbool(var);
> > + if (!strcmp(value, "fsync"))
> > + fsync_method = FSYNC_METHOD_FSYNC;
> > + else if (!strcmp(value, "writeout-only"))
> > + fsync_method = FSYNC_METHOD_WRITEOUT_ONLY;
> > + else
>
> As a non-nit comment I think this config schema looks great so far.
>
> > + warning(_("unknown %s value '%s'"), var, value);
>
> Just a suggestion maybe something slightly scarier like:
>
> "unknown core.fsyncMethod value '%s'; config from future git version? ignoring requested fsync strategy"
>
> Also using the nicer camelCased version instead of "var" (also helps
> translators with context...)
>
Will fix. The motivation for this scheme was to 'factor' the messages
so there would be less to translate. But I see now that the message
doesn't have enough context to translate reasonably.
> > +int git_fsync(int fd, enum fsync_action action)
> > +{
> > + switch (action) {
> > + case FSYNC_WRITEOUT_ONLY:
> > +
> > +#ifdef __APPLE__
> > + /*
> > + * on macOS, fsync just causes filesystem cache writeback but does not
> > + * flush hardware caches.
> > + */
> > + return fsync(fd);
> > +#endif
> > +
> > +#ifdef HAVE_SYNC_FILE_RANGE
> > + /*
> > + * On linux 2.6.17 and above, sync_file_range is the way to issue
> > + * a writeback without a hardware flush. An offset of 0 and size of 0
> > + * indicates writeout of the entire file and the wait flags ensure that all
> > + * dirty data is written to the disk (potentially in a disk-side cache)
> > + * before we continue.
> > + */
> > +
> > + return sync_file_range(fd, 0, 0, SYNC_FILE_RANGE_WAIT_BEFORE |
> > + SYNC_FILE_RANGE_WRITE |
> > + SYNC_FILE_RANGE_WAIT_AFTER);
> > +#endif
> > +
> > +#ifdef fsync_no_flush
> > + return fsync_no_flush(fd);
> > +#endif
> > +
> > + errno = ENOSYS;
> > + return -1;
> > +
> > + case FSYNC_HARDWARE_FLUSH:
> > + /*
> > + * On some platforms fsync may return EINTR. Try again in this
> > + * case, since callers asking for a hardware flush may die if
> > + * this function returns an error.
> > + */
> > + for (;;) {
> > + int err;
> > +#ifdef __APPLE__
> > + err = fcntl(fd, F_FULLFSYNC);
> > +#else
> > + err = fsync(fd);
> > +#endif
> > + if (err >= 0 || errno != EINTR)
> > + return err;
> > + }
> > +
> > + default:
> > + BUG("unexpected git_fsync(%d) call", action);
>
> Don't include such "default" cases, you have an exhaustive "enum", if
> you skip it the compiler will check this for you.
>
Junio gave the feedback to include this "default:" case in the switch
[1]. Removing the default leads to the "error: control reaches end of
non-void function" on gcc. Fixing that error and adding a trial option
does give the exhaustiveness error that you're talking about. I'd
rather just leave this as-is since the BUG() obviates the need for an
in-practice-unreachable return statement.
[1] https://lore.kernel.org/git/xmqqfsu70x58.fsf@gitster.g/
> > + }
> > +}
> > +
> > static int warn_if_unremovable(const char *op, const char *file, int rc)
>
> Just a code nit: I think it's very much preferred if possible to have as
> much of code like this compile on all platforms. See the series at
> 4002e87cb25 (grep: remove #ifdef NO_PTHREADS, 2018-11-03) is part of for
> a good example.
>
> Maybe not worth it in this case since they're not nested ifdef's.
>
> I'm basically thinking of something (also re Patrick's comment on the
> 2nd patch) where we have a platform_fsync() whose return
> value/arguments/whatever capture this "I want to return now" or "you
> should be looping" and takes the enum_fsync_action" strategy.
>
> Then the git_fsync() would be the platform-independent looping etc., and
> another funciton would do the "one fsync at a time, maybe call me
> again".
>
> Maybe it would suck more, just food for thought... :)
I'm going to introduce a new static function called fsync_loop which
does the looping and I'll call it from git_fsync. That appears to be
the cleanest to me to address your and Patrick's feedback.
Thanks,
Neeraj
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v2 0/3] A design for future-proofing fsync() configuration
2021-12-07 11:56 ` [PATCH v2 0/3] A design for future-proofing fsync() configuration Patrick Steinhardt
@ 2021-12-08 0:44 ` Neeraj Singh
0 siblings, 0 replies; 122+ messages in thread
From: Neeraj Singh @ 2021-12-08 0:44 UTC (permalink / raw)
To: Patrick Steinhardt
Cc: Neeraj K. Singh via GitGitGadget, Git List, Randall S. Becker,
Bagas Sanjaya, Elijah Newren,
Ævar Arnfjörð Bjarmason, Neeraj K. Singh
On Tue, Dec 7, 2021 at 3:57 AM Patrick Steinhardt <ps@pks.im> wrote:
> While I bail from the question of whether we want to grant as much
> configurability to the user as this patch series does, I quite like the
> implementation. It feels rather straight-forward and it's easy to see
> how to extend it to support syncing of other subsystems like the loose
> refs.
>
> Thanks!
>
> Patrick
Thanks for the positive comment. I'm assuming that a major Git
services like GitHub or GitLab would be able to take advantage of the
granular options and knowledge of their hosting environment to choose
the right values for any server-side git deployments. I'd probably
turn off syncing for derived stuff like the commit-graph file and pack
metadata.
My underlying interest in all of these changes is to make Windows stop
looking so bad (we're defaulting to core.fsyncobjectfiles=true).
Batch mode should give similar safety and much more optimizable
performance in our environment.
Thanks,
Neeraj
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v2 2/3] core.fsync: introduce granular fsync control
2021-12-07 21:44 ` Neeraj Singh
@ 2021-12-08 10:05 ` Ævar Arnfjörð Bjarmason
2021-12-09 0:14 ` Neeraj Singh
0 siblings, 1 reply; 122+ messages in thread
From: Ævar Arnfjörð Bjarmason @ 2021-12-08 10:05 UTC (permalink / raw)
To: Neeraj Singh
Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker,
Bagas Sanjaya, Elijah Newren, Patrick Steinhardt, Neeraj Singh
On Tue, Dec 07 2021, Neeraj Singh wrote:
> On Tue, Dec 7, 2021 at 5:01 AM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
>>
>>
>> On Tue, Dec 07 2021, Neeraj Singh via GitGitGadget wrote:
>>
>> > From: Neeraj Singh <neerajsi@microsoft.com>
>> >
>> > This commit introduces the `core.fsync` configuration
>> > knob which can be used to control how components of the
>> > repository are made durable on disk.
>> >
>> > This setting allows future extensibility of the list of
>> > syncable components:
>> > * We issue a warning rather than an error for unrecognized
>> > components, so new configs can be used with old Git versions.
>>
>> Looks good!
>>
>> > * We support negation, so users can choose one of the default
>> > aggregate options and then remove components that they don't
>> > want. The user would then harden any new components added in
>> > a Git version update.
>>
>> I think this config schema makes sense, but just a (I think important)
>> comment on the "how" not "what" of it. It's really much better to define
>> config as:
>>
>> [some]
>> key = value
>> key = value2
>>
>> Than:
>>
>> [some]
>> key = value,value2
>>
>> The reason is that "git config" has good support for working with
>> multi-valued stuff, so you can do e.g.:
>>
>> git config --get-all -z some.key
>>
>> And you can easily (re)set such config e.g. with --replace-all etc., but
>> for comma-delimited you (and users) need to do all that work themselves.
>>
>> Similarly instead of:
>>
>> some.key = want-this
>> some.key = -not-this
>> some.key = but-want-this
>>
>> I think it's better to just have two lists, one inclusive another
>> exclusive. E.g. see "log.decorate" and "log.excludeDecoration",
>> "transfer.hideRefs"
>>
>> Which would mean:
>>
>> core.fsync = want-this
>> core.fsyncExcludes = -not-this
>>
>> For some value of "fsyncExcludes", maybe "noFsync"? Anyway, just a
>> suggestion on making this easier for users & the implementation.
>>
>
> Maybe there's some way to handle this I'm unaware of, but a
> disadvantage of your multi-valued config proposal is that it's harder,
> for example, for a per-repo config store to reasonably override a
> per-user config store. With the configuration scheme as-is, I can
> have a per-user setting like `core.fsync=all` which covers my typical
> repos, but then have a maintainer repo with a private setting of
> `core.fsync=none` to speed up cases where I'm mostly working with
> other people's changes that are backed up in email or server-side
> repos. The latter setting conveniently overrides the former setting
> in all aspects.
Even if you turn just your comma-delimited proposal into a list proposal
can't we just say that the last one wins? Then it won't matter what cmae
before, you'd specify "core.fsync=none" in your local .git/config.
But this is also a general issue with a bunch of things in git's config
space. I'd rather see us use the list-based values and just come up with
some general way to reset them that works for all keys, rather than
regretting having comma-delimited values that'll be harder to work with
& parse, which will be a legacy wart if/when we come up with a way to
say "reset all previous settings".
> Also, with the core.fsync and core.fsyncExcludes, how would you spell
> "don't sync anything"? Would you still have the aggregate options.?
core.fsyncExcludes = *
I.e. the same as the core.fsync=none above, anyway I still like the
wildcard thing below a bit more...
>> > This also supports the common request of doing absolutely no
>> > fysncing with the `core.fsync=none` value, which is expected
>> > to make the test suite faster.
>>
>> Let's just use the git_parse_maybe_bool() or git_parse_maybe_bool_text()
>> so we'll accept "false", "off", "no" like most other such config?
>
> Junio's previous feedback when discussing batch mode [1] was to offer
> less flexibility when parsing new values of these configuration
> options. I agree with his statement that "making machine-readable
> tokens be spelled in different ways is a 'disease'." I'd like to
> leave this as-is so that the documentation can clearly state the exact
> set of allowable values.
>
> [1] https://lore.kernel.org/git/xmqqr1dqzyl7.fsf@gitster.g/
I think he's talking about batch, Batch, BATCH, bAtCh etc. there. But
the "maybe bool" is a stanard pattern we use.
I don't think we'd call one of these 0, off, no or false etc. to avoid
confusion, so then you can use git_parse_maybe_...()
>>
>> > Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
>> > ---
>> > Documentation/config/core.txt | 27 +++++++++----
>> > builtin/fast-import.c | 2 +-
>> > builtin/index-pack.c | 4 +-
>> > builtin/pack-objects.c | 8 ++--
>> > bulk-checkin.c | 5 ++-
>> > cache.h | 39 +++++++++++++++++-
>> > commit-graph.c | 3 +-
>> > config.c | 76 ++++++++++++++++++++++++++++++++++-
>> > csum-file.c | 5 ++-
>> > csum-file.h | 3 +-
>> > environment.c | 1 +
>> > midx.c | 3 +-
>> > object-file.c | 3 +-
>> > pack-bitmap-write.c | 3 +-
>> > pack-write.c | 13 +++---
>> > read-cache.c | 2 +-
>> > 16 files changed, 164 insertions(+), 33 deletions(-)
>> >
>> > diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
>> > index dbb134f7136..4f1747ec871 100644
>> > --- a/Documentation/config/core.txt
>> > +++ b/Documentation/config/core.txt
>> > @@ -547,6 +547,25 @@ core.whitespace::
>> > is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent`
>> > errors. The default tab width is 8. Allowed values are 1 to 63.
>> >
>> > +core.fsync::
>> > + A comma-separated list of parts of the repository which should be
>> > + hardened via the core.fsyncMethod when created or modified. You can
>> > + disable hardening of any component by prefixing it with a '-'. Later
>> > + items take precedence over earlier ones in the list. For example,
>> > + `core.fsync=all,-pack-metadata` means "harden everything except pack
>> > + metadata." Items that are not hardened may be lost in the event of an
>> > + unclean system shutdown.
>> > ++
>> > +* `none` disables fsync completely. This must be specified alone.
>> > +* `loose-object` hardens objects added to the repo in loose-object form.
>> > +* `pack` hardens objects added to the repo in packfile form.
>> > +* `pack-metadata` hardens packfile bitmaps and indexes.
>> > +* `commit-graph` hardens the commit graph file.
>> > +* `objects` is an aggregate option that includes `loose-objects`, `pack`,
>> > + `pack-metadata`, and `commit-graph`.
>> > +* `default` is an aggregate option that is equivalent to `objects,-loose-object`
>> > +* `all` is an aggregate option that syncs all individual components above.
>> > +
>>
>> It's probably a *bit* more work to set up, but I wonder if this wouldn't
>> be simpler if we just said (and this is partially going against what I
>> noted above):
>>
>> == BEGIN DOC
>>
>> core.fsync is a multi-value config variable where each item is a
>> pathspec that'll get matched the same way as 'git-ls-files' et al.
>>
>> When we sync pretend that a path like .git/objects/de/adbeef... is
>> relative to the top-level of the git
>> directory. E.g. "objects/de/adbeaf.." or "objects/pack/...".
>>
>> You can then supply a list of wildcards and exclusions to configure
>> syncing. or "false", "off" etc. to turn it off. These are synonymous
>> with:
>>
>> ; same as "false"
>> core.fsync = ":!*"
>>
>> Or:
>>
>> ; same as "true"
>> core.fsync = "*"
>>
>> Or, to selectively sync some things and not others:
>>
>> ;; Sync objects, but not "info"
>> core.fsync = ":!objects/info/**"
>> core.fsync = "objects/**"
>>
>> See gitrepository-layout(5) for details about what sort of paths you
>> might be expected to match. Not all paths listed there will go through
>> this mechanism (e.g. currently objects do, but nothing to do with config
>> does).
>>
>> We can and will match this against "fake paths", e.g. when writing out
>> packs we may match against just the string "objects/pack", we're not
>> going to re-check if every packfile we're writing matches your globs,
>> ditto for loose objects. Be reasonable!
>>
>> This metharism is intended as a shorthand that provides some flexibility
>> when fsyncing, while not forcing git to come up with labels for all
>> paths the git dir, or to support crazyness like "objects/de/adbeef*"
>>
>> More paths may be added or removed in the future, and we make no
>> promises that we won't move things around, so if in doubt use
>> e.g. "true" or a wide pattern match like "objects/**". When in doubt
>> stick to the golden path of examples provided in this documentation.
>>
>> == END DOC
>>
>>
>> It's a tad more complex to set up, but I wonder if that isn't worth
>> it. It nicely gets around any current and future issues of deciding what
>> labels such as "loose-object" etc. to pick, as well as slotting into an
>> existing method of doing exclude/include lists.
>>
>
> I think this proposal is a lot of complexity to avoid coming up with a
> new name for syncable things as they are added to Git. A path based
> mechanism makes it hard to document for the (advanced) user what the
> full set of things is and how it might change from release to release.
> I think the current core.fsync scheme is a bit easier to understand,
> query, and extend.
We document it in gitrepository-layout(5). Yeah it has some
disadvantages, but one advantage is that you could make the
composability easy. I.e. if last exclude wins then a setting of:
core.fsync = ":!*"
core.fsync = "objects/**"
Would reset all previous matches & only match objects/**.
>> > diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
>> > index 857be7826f3..916c55d6ce9 100644
>> > --- a/builtin/pack-objects.c
>> > +++ b/builtin/pack-objects.c
>> > @@ -1204,11 +1204,13 @@ static void write_pack_file(void)
>> > * If so, rewrite it like in fast-import
>> > */
>> > if (pack_to_stdout) {
>> > - finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_CLOSE);
>> > + finalize_hashfile(f, hash, FSYNC_COMPONENT_NONE,
>> > + CSUM_HASH_IN_STREAM | CSUM_CLOSE);
>>
>> Not really related to this per-se, but since you're touching the API
>> everything goes through I wonder if callers should just always try to
>> fsync, and we can just catch EROFS and EINVAL in the wrapper if someone
>> tries to flush stdout, or catch the fd at that lower level.
>>
>> Or maybe there's a good reason for this...
>
> It's platform dependent, but I'd expect fsync would do something for
> pipes or stdout redirected to a file. In these cases we really don't
> want to fsync since we have no idea what we're talking to and we're
> potentially worsening performance for probably no benefit.
Yeah maybe we should just leave it be.
I'd think the C library returning EINVAL would be a trivial performance
cost though.
It just seemed odd to hardcode assumptions about what can and can't be
synced when the POSIX defined function will also tell us that.
Anyway...
>> > [...]
>> > +/*
>> > + * These values are used to help identify parts of a repository to fsync.
>> > + * FSYNC_COMPONENT_NONE identifies data that will not be a persistent part of the
>> > + * repository and so shouldn't be fsynced.
>> > + */
>> > +enum fsync_component {
>> > + FSYNC_COMPONENT_NONE = 0,
>>
>> I haven't read ahead much but in most other such cases we don't define
>> the "= 0", just start at 1<<0, then check the flags elsewhere...
>>
>> > +static const struct fsync_component_entry {
>> > + const char *name;
>> > + enum fsync_component component_bits;
>> > +} fsync_component_table[] = {
>> > + { "loose-object", FSYNC_COMPONENT_LOOSE_OBJECT },
>> > + { "pack", FSYNC_COMPONENT_PACK },
>> > + { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA },
>> > + { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH },
>> > + { "objects", FSYNC_COMPONENTS_OBJECTS },
>> > + { "default", FSYNC_COMPONENTS_DEFAULT },
>> > + { "all", FSYNC_COMPONENTS_ALL },
>> > +};
>> > +
>> > +static enum fsync_component parse_fsync_components(const char *var, const char *string)
>> > +{
>> > + enum fsync_component output = 0;
>> > +
>> > + if (!strcmp(string, "none"))
>> > + return output;
>> > +
>> > + while (string) {
>> > + int i;
>> > + size_t len;
>> > + const char *ep;
>> > + int negated = 0;
>> > + int found = 0;
>> > +
>> > + string = string + strspn(string, ", \t\n\r");
>>
>> Aside from the "use a list" isn't this hardcoding some windows-specific
>> assumptions with \n\r? Maybe not...
>
> I shamelessly stole this code from parse_whitespace_rule. I thought
> about making a helper to be called by both functions, but the amount
> of state going into and out of the wrapper via arguments was
> substantial and seemed to negate the benefit of deduplication.
FWIW string_list_split() is easier to work with in those cases, or at
least I think so...
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v2 2/3] core.fsync: introduce granular fsync control
2021-12-08 10:05 ` Ævar Arnfjörð Bjarmason
@ 2021-12-09 0:14 ` Neeraj Singh
2021-12-09 0:44 ` Junio C Hamano
2021-12-09 4:08 ` Ævar Arnfjörð Bjarmason
0 siblings, 2 replies; 122+ messages in thread
From: Neeraj Singh @ 2021-12-09 0:14 UTC (permalink / raw)
To: Ævar Arnfjörð Bjarmason
Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker,
Bagas Sanjaya, Elijah Newren, Patrick Steinhardt, Neeraj Singh
On Wed, Dec 8, 2021 at 2:17 AM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
>
>
> On Tue, Dec 07 2021, Neeraj Singh wrote:
>
> > On Tue, Dec 7, 2021 at 5:01 AM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
> >>
> >>
> >> On Tue, Dec 07 2021, Neeraj Singh via GitGitGadget wrote:
> >>
> >> > From: Neeraj Singh <neerajsi@microsoft.com>
> >> >
> >> > This commit introduces the `core.fsync` configuration
> >> > knob which can be used to control how components of the
> >> > repository are made durable on disk.
> >> >
> >> > This setting allows future extensibility of the list of
> >> > syncable components:
> >> > * We issue a warning rather than an error for unrecognized
> >> > components, so new configs can be used with old Git versions.
> >>
> >> Looks good!
> >>
> >> > * We support negation, so users can choose one of the default
> >> > aggregate options and then remove components that they don't
> >> > want. The user would then harden any new components added in
> >> > a Git version update.
> >>
> >> I think this config schema makes sense, but just a (I think important)
> >> comment on the "how" not "what" of it. It's really much better to define
> >> config as:
> >>
> >> [some]
> >> key = value
> >> key = value2
> >>
> >> Than:
> >>
> >> [some]
> >> key = value,value2
> >>
> >> The reason is that "git config" has good support for working with
> >> multi-valued stuff, so you can do e.g.:
> >>
> >> git config --get-all -z some.key
> >>
> >> And you can easily (re)set such config e.g. with --replace-all etc., but
> >> for comma-delimited you (and users) need to do all that work themselves.
> >>
> >> Similarly instead of:
> >>
> >> some.key = want-this
> >> some.key = -not-this
> >> some.key = but-want-this
> >>
> >> I think it's better to just have two lists, one inclusive another
> >> exclusive. E.g. see "log.decorate" and "log.excludeDecoration",
> >> "transfer.hideRefs"
> >>
> >> Which would mean:
> >>
> >> core.fsync = want-this
> >> core.fsyncExcludes = -not-this
> >>
> >> For some value of "fsyncExcludes", maybe "noFsync"? Anyway, just a
> >> suggestion on making this easier for users & the implementation.
> >>
> >
> > Maybe there's some way to handle this I'm unaware of, but a
> > disadvantage of your multi-valued config proposal is that it's harder,
> > for example, for a per-repo config store to reasonably override a
> > per-user config store. With the configuration scheme as-is, I can
> > have a per-user setting like `core.fsync=all` which covers my typical
> > repos, but then have a maintainer repo with a private setting of
> > `core.fsync=none` to speed up cases where I'm mostly working with
> > other people's changes that are backed up in email or server-side
> > repos. The latter setting conveniently overrides the former setting
> > in all aspects.
>
> Even if you turn just your comma-delimited proposal into a list proposal
> can't we just say that the last one wins? Then it won't matter what cmae
> before, you'd specify "core.fsync=none" in your local .git/config.
>
> But this is also a general issue with a bunch of things in git's config
> space. I'd rather see us use the list-based values and just come up with
> some general way to reset them that works for all keys, rather than
> regretting having comma-delimited values that'll be harder to work with
> & parse, which will be a legacy wart if/when we come up with a way to
> say "reset all previous settings".
>
> > Also, with the core.fsync and core.fsyncExcludes, how would you spell
> > "don't sync anything"? Would you still have the aggregate options.?
>
> core.fsyncExcludes = *
>
> I.e. the same as the core.fsync=none above, anyway I still like the
> wildcard thing below a bit more...
I'm not going to take this feedback unless there are additional votes
from the Git community in this direction. I make the claim that
single-valued comma-separated config lists are easier to work with in
the existing Git infrastructure. We already use essentially the same
parsing code for the core.whitespace variable and users are used to
this syntax there. There are several other comma-separated lists in
the config space, so this construct has precedence and will be with
Git for some time. Also, fsync configurations aren't composable like
some other configurations may be. It makes sense to have a holistic
singular fsync configuration, which is best represented by a single
variable.
> >> > This also supports the common request of doing absolutely no
> >> > fysncing with the `core.fsync=none` value, which is expected
> >> > to make the test suite faster.
> >>
> >> Let's just use the git_parse_maybe_bool() or git_parse_maybe_bool_text()
> >> so we'll accept "false", "off", "no" like most other such config?
> >
> > Junio's previous feedback when discussing batch mode [1] was to offer
> > less flexibility when parsing new values of these configuration
> > options. I agree with his statement that "making machine-readable
> > tokens be spelled in different ways is a 'disease'." I'd like to
> > leave this as-is so that the documentation can clearly state the exact
> > set of allowable values.
> >
> > [1] https://lore.kernel.org/git/xmqqr1dqzyl7.fsf@gitster.g/
>
> I think he's talking about batch, Batch, BATCH, bAtCh etc. there. But
> the "maybe bool" is a stanard pattern we use.
>
> I don't think we'd call one of these 0, off, no or false etc. to avoid
> confusion, so then you can use git_parse_maybe_...()
I don't see the advantage of having multiple ways of specifying
"none". The user can read the doc and know exactly what to write. If
they write something unallowable, they get a clear warning and they
can read the doc again to figure out what to write. This isn't a
boolean options at all, so why should we entertain bool-like ways of
spelling it?
> >>
> >> > Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
> >> > ---
> >> > Documentation/config/core.txt | 27 +++++++++----
> >> > builtin/fast-import.c | 2 +-
> >> > builtin/index-pack.c | 4 +-
> >> > builtin/pack-objects.c | 8 ++--
> >> > bulk-checkin.c | 5 ++-
> >> > cache.h | 39 +++++++++++++++++-
> >> > commit-graph.c | 3 +-
> >> > config.c | 76 ++++++++++++++++++++++++++++++++++-
> >> > csum-file.c | 5 ++-
> >> > csum-file.h | 3 +-
> >> > environment.c | 1 +
> >> > midx.c | 3 +-
> >> > object-file.c | 3 +-
> >> > pack-bitmap-write.c | 3 +-
> >> > pack-write.c | 13 +++---
> >> > read-cache.c | 2 +-
> >> > 16 files changed, 164 insertions(+), 33 deletions(-)
> >> >
> >> > diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
> >> > index dbb134f7136..4f1747ec871 100644
> >> > --- a/Documentation/config/core.txt
> >> > +++ b/Documentation/config/core.txt
> >> > @@ -547,6 +547,25 @@ core.whitespace::
> >> > is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent`
> >> > errors. The default tab width is 8. Allowed values are 1 to 63.
> >> >
> >> > +core.fsync::
> >> > + A comma-separated list of parts of the repository which should be
> >> > + hardened via the core.fsyncMethod when created or modified. You can
> >> > + disable hardening of any component by prefixing it with a '-'. Later
> >> > + items take precedence over earlier ones in the list. For example,
> >> > + `core.fsync=all,-pack-metadata` means "harden everything except pack
> >> > + metadata." Items that are not hardened may be lost in the event of an
> >> > + unclean system shutdown.
> >> > ++
> >> > +* `none` disables fsync completely. This must be specified alone.
> >> > +* `loose-object` hardens objects added to the repo in loose-object form.
> >> > +* `pack` hardens objects added to the repo in packfile form.
> >> > +* `pack-metadata` hardens packfile bitmaps and indexes.
> >> > +* `commit-graph` hardens the commit graph file.
> >> > +* `objects` is an aggregate option that includes `loose-objects`, `pack`,
> >> > + `pack-metadata`, and `commit-graph`.
> >> > +* `default` is an aggregate option that is equivalent to `objects,-loose-object`
> >> > +* `all` is an aggregate option that syncs all individual components above.
> >> > +
> >>
> >> It's probably a *bit* more work to set up, but I wonder if this wouldn't
> >> be simpler if we just said (and this is partially going against what I
> >> noted above):
> >>
> >> == BEGIN DOC
> >>
> >> core.fsync is a multi-value config variable where each item is a
> >> pathspec that'll get matched the same way as 'git-ls-files' et al.
> >>
> >> When we sync pretend that a path like .git/objects/de/adbeef... is
> >> relative to the top-level of the git
> >> directory. E.g. "objects/de/adbeaf.." or "objects/pack/...".
> >>
> >> You can then supply a list of wildcards and exclusions to configure
> >> syncing. or "false", "off" etc. to turn it off. These are synonymous
> >> with:
> >>
> >> ; same as "false"
> >> core.fsync = ":!*"
> >>
> >> Or:
> >>
> >> ; same as "true"
> >> core.fsync = "*"
> >>
> >> Or, to selectively sync some things and not others:
> >>
> >> ;; Sync objects, but not "info"
> >> core.fsync = ":!objects/info/**"
> >> core.fsync = "objects/**"
> >>
> >> See gitrepository-layout(5) for details about what sort of paths you
> >> might be expected to match. Not all paths listed there will go through
> >> this mechanism (e.g. currently objects do, but nothing to do with config
> >> does).
> >>
> >> We can and will match this against "fake paths", e.g. when writing out
> >> packs we may match against just the string "objects/pack", we're not
> >> going to re-check if every packfile we're writing matches your globs,
> >> ditto for loose objects. Be reasonable!
> >>
> >> This metharism is intended as a shorthand that provides some flexibility
> >> when fsyncing, while not forcing git to come up with labels for all
> >> paths the git dir, or to support crazyness like "objects/de/adbeef*"
> >>
> >> More paths may be added or removed in the future, and we make no
> >> promises that we won't move things around, so if in doubt use
> >> e.g. "true" or a wide pattern match like "objects/**". When in doubt
> >> stick to the golden path of examples provided in this documentation.
> >>
> >> == END DOC
> >>
> >>
> >> It's a tad more complex to set up, but I wonder if that isn't worth
> >> it. It nicely gets around any current and future issues of deciding what
> >> labels such as "loose-object" etc. to pick, as well as slotting into an
> >> existing method of doing exclude/include lists.
> >>
> >
> > I think this proposal is a lot of complexity to avoid coming up with a
> > new name for syncable things as they are added to Git. A path based
> > mechanism makes it hard to document for the (advanced) user what the
> > full set of things is and how it might change from release to release.
> > I think the current core.fsync scheme is a bit easier to understand,
> > query, and extend.
>
> We document it in gitrepository-layout(5). Yeah it has some
> disadvantages, but one advantage is that you could make the
> composability easy. I.e. if last exclude wins then a setting of:
>
> core.fsync = ":!*"
> core.fsync = "objects/**"
>
> Would reset all previous matches & only match objects/**.
>
The value of changing this is predicated on taking your previous
multi-valued config proposal, which I'm still not at all convinced
about. The schema in the current (v1-v2) version of the patch already
includes an example of extending the list of syncable things, and
Patrick Steinhardt made it clear that he feels comfortable adding
'refs' to the same schema in a future change.
I'll also emphasize that we're talking about a non-functional,
relatively corner-case behavioral configuration. These values don't
change how git's interface behaves except when the system crashes
during a git command or shortly after one completes.
While you may not personally love the proposed configuration
interface, I'd want your view on some questions:
1. Is it easy for the (advanced) user to set a configuration?
2. Is it easy for the (advanced) user to see what was configured?
3. Is it easy for the Git community to build on this as we want to add
things to the list of things to sync?
a) Is there a good best practice configuration so that people can
avoid losing integrity for new stuff that they are intending to sync.
b) If someone has a custom configuration, can that custom
configuration do something reasonable as they upgrade versions of Git?
** In response to this question, I might see some value
in adding a 'derived-metadata' aggregate that can be disabled so that
a custom configuration can exclude those as they change version to
version.
c) Is it too much maintenance overhead to consider how to present
this configuration knob for any new hashfile or other datafile in the
git repo?
4. Is there a good path forward to change the default syncable set,
both in git-for-windows and in Git for other platforms?
> >> > diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
> >> > index 857be7826f3..916c55d6ce9 100644
> >> > --- a/builtin/pack-objects.c
> >> > +++ b/builtin/pack-objects.c
> >> > @@ -1204,11 +1204,13 @@ static void write_pack_file(void)
> >> > * If so, rewrite it like in fast-import
> >> > */
> >> > if (pack_to_stdout) {
> >> > - finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_CLOSE);
> >> > + finalize_hashfile(f, hash, FSYNC_COMPONENT_NONE,
> >> > + CSUM_HASH_IN_STREAM | CSUM_CLOSE);
> >>
> >> Not really related to this per-se, but since you're touching the API
> >> everything goes through I wonder if callers should just always try to
> >> fsync, and we can just catch EROFS and EINVAL in the wrapper if someone
> >> tries to flush stdout, or catch the fd at that lower level.
> >>
> >> Or maybe there's a good reason for this...
> >
> > It's platform dependent, but I'd expect fsync would do something for
> > pipes or stdout redirected to a file. In these cases we really don't
> > want to fsync since we have no idea what we're talking to and we're
> > potentially worsening performance for probably no benefit.
>
> Yeah maybe we should just leave it be.
>
> I'd think the C library returning EINVAL would be a trivial performance
> cost though.
>
> It just seemed odd to hardcode assumptions about what can and can't be
> synced when the POSIX defined function will also tell us that.
>
Redirecting stdout to a file seems like a common usage for this
command. That would definitely be fsyncable, but Git has no idea what
its proper category is since there's no way to know the purpose or
lifetime of the packfile. I'm going to leave this be, because I'd
posit that "can it be fsynced?" is not the same as "should it be
fsynced?". The latter question can't be answered for stdout.
>
> >> > [...]
> >> > +/*
> >> > + * These values are used to help identify parts of a repository to fsync.
> >> > + * FSYNC_COMPONENT_NONE identifies data that will not be a persistent part of the
> >> > + * repository and so shouldn't be fsynced.
> >> > + */
> >> > +enum fsync_component {
> >> > + FSYNC_COMPONENT_NONE = 0,
> >>
> >> I haven't read ahead much but in most other such cases we don't define
> >> the "= 0", just start at 1<<0, then check the flags elsewhere...
> >>
> >> > +static const struct fsync_component_entry {
> >> > + const char *name;
> >> > + enum fsync_component component_bits;
> >> > +} fsync_component_table[] = {
> >> > + { "loose-object", FSYNC_COMPONENT_LOOSE_OBJECT },
> >> > + { "pack", FSYNC_COMPONENT_PACK },
> >> > + { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA },
> >> > + { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH },
> >> > + { "objects", FSYNC_COMPONENTS_OBJECTS },
> >> > + { "default", FSYNC_COMPONENTS_DEFAULT },
> >> > + { "all", FSYNC_COMPONENTS_ALL },
> >> > +};
> >> > +
> >> > +static enum fsync_component parse_fsync_components(const char *var, const char *string)
> >> > +{
> >> > + enum fsync_component output = 0;
> >> > +
> >> > + if (!strcmp(string, "none"))
> >> > + return output;
> >> > +
> >> > + while (string) {
> >> > + int i;
> >> > + size_t len;
> >> > + const char *ep;
> >> > + int negated = 0;
> >> > + int found = 0;
> >> > +
> >> > + string = string + strspn(string, ", \t\n\r");
> >>
> >> Aside from the "use a list" isn't this hardcoding some windows-specific
> >> assumptions with \n\r? Maybe not...
> >
> > I shamelessly stole this code from parse_whitespace_rule. I thought
> > about making a helper to be called by both functions, but the amount
> > of state going into and out of the wrapper via arguments was
> > substantial and seemed to negate the benefit of deduplication.
>
> FWIW string_list_split() is easier to work with in those cases, or at
> least I think so...
This code runs at startup for a variable that may be present on some
installations. The nice property of the current patch's code is that
it's already a well-tested pattern that doesn't do any allocations as
it's working, unlike string_list_split().
I hope you know that I appreciate your review feedback, even though
I'm pushing back on most of it so far this round. I'll be sending v3
to the list soon after giving it another look over.
Thanks,
Neeraj
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v2 2/3] core.fsync: introduce granular fsync control
2021-12-09 0:14 ` Neeraj Singh
@ 2021-12-09 0:44 ` Junio C Hamano
2021-12-09 4:08 ` Ævar Arnfjörð Bjarmason
1 sibling, 0 replies; 122+ messages in thread
From: Junio C Hamano @ 2021-12-09 0:44 UTC (permalink / raw)
To: Neeraj Singh
Cc: Ævar Arnfjörð Bjarmason,
Neeraj Singh via GitGitGadget, Git List, Randall S. Becker,
Bagas Sanjaya, Elijah Newren, Patrick Steinhardt, Neeraj Singh
Neeraj Singh <nksingh85@gmail.com> writes:
> I'm not going to take this feedback unless there are additional votes
> from the Git community in this direction. I make the claim that
> single-valued comma-separated config lists are easier to work with in
> the existing Git infrastructure. We already use essentially the same
> parsing code for the core.whitespace variable and users are used to
> this syntax there. There are several other comma-separated lists in
> the config space, so this construct has precedence and will be with
> Git for some time. Also, fsync configurations aren't composable like
> some other configurations may be. It makes sense to have a holistic
> singular fsync configuration, which is best represented by a single
> variable.
I haven't caught up with the discussion in this thread, even though
I have been meaning to think about it for some time---I just haven't
got around to it (sorry). So I'll stop at giving a general guidance
and leave the decision if it applies to this particular discussion
to readers.
As the inventor of core.whitespace "list of values and its parser, I
am biased, but I would say that it works well for simple things that
do not need too much overriding. The other side of the coin is that
it can become very awkward going forward if we use it to things that
have more complex needs than answering a simple question like "what
whitespace errors should be checked?".
More specifically, core.whitespace is pecuriar in a few ways.
* It does follow the usual "the last one wins" rule, but in a
strange way. Notice the "unsigned rule = WS_DEFAULT_RULE"
assignment at the beginning of ws.c::parse_whitespace_rule()?
For each configuration "core.whitespace=<list>" we encounter,
we start from the default, discarding everything we saw so far,
and tweak that default value with tokens found on the list.
* You cannot do "The system config gives one set of values, which
you tweak with the personal config, which is further tweaked with
the repository config" as the consequence. This is blessing and
is curse at the same time, as it makes inspection simpler when
things go wrong (you only need to check whta the last one does),
but it is harder to share the common things in more common file.
* Its design relies on the choices being strings chosen from a
fixed vocabulary to allow you to say "the value of the
configuration variable is a list of tokens separated by a comma"
and "the default has X bit set, but we disable that with -X".
For a configuration variable whose value is an arbitrary string
or a number, obviously that approach would not work.
If the need of the topic is simple enough that the above limitation
core.whitespace does not pose a problem going forward, it would be
fine, but we may regret the choice we make today if that is not the
case.
Thanks.
^ permalink raw reply [flat|nested] 122+ messages in thread
* [PATCH v3 0/4] A design for future-proofing fsync() configuration
2021-12-07 2:46 ` [PATCH v2 0/3] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget
` (3 preceding siblings ...)
2021-12-07 11:56 ` [PATCH v2 0/3] A design for future-proofing fsync() configuration Patrick Steinhardt
@ 2021-12-09 0:57 ` Neeraj K. Singh via GitGitGadget
2021-12-09 0:57 ` [PATCH v3 1/4] core.fsyncmethod: add writeout-only mode Neeraj Singh via GitGitGadget
` (5 more replies)
4 siblings, 6 replies; 122+ messages in thread
From: Neeraj K. Singh via GitGitGadget @ 2021-12-09 0:57 UTC (permalink / raw)
To: git; +Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps,
Neeraj K. Singh
This is an implementation of an extensible configuration mechanism for
fsyncing persistent components of a repo.
The main goals are to separate the "what" to sync from the "how". There are
now two settings: core.fsync - Control the 'what', including the index.
core.fsyncMethod - Control the 'how'. Currently we support writeout-only and
full fsync.
Syncing of refs can be layered on top of core.fsync. And batch mode will be
layered on core.fsyncMethod.
core.fsyncobjectfiles is removed and will issue a deprecation warning if
it's seen.
I'd like to get agreement on this direction before submitting batch mode to
the list. The batch mode series is available to view at
https://github.com/neerajsi-msft/git/pull/1.
Please see [1], [2], and [3] for discussions that led to this series.
One major concern I'd voice that is adverse to this change: when a new
persistent file is added to the Git repo, the person adding that file will
need to update this configuration code and the documentation. Maybe this is
the right thing to always think about, but the FSYNC_COMPONENT lists will be
a single place that will receive a number of updates over time.
Note: There's a minor conflict with ns/tmp-objdir. In
object-file.c:close_loose_object we need to resolve it like this:
if (!the_repository->objects->odb->will_destroy)
fsync_component_or_die(FSYNC_COMPONENT_LOOSE_OBJECT, fd, "loose object
file");
V3 changes:
* Remove relative path from git-compat-util.h include [4].
* Updated newly added warning texts to have more context for localization
[4].
* Fixed tab spacing in enum fsync_action
* Moved the fsync looping out to a helper and do it consistently. [4]
* Changed commit description to use camelCase for config names. [5]
* Add an optional fourth patch with derived-metadata so that the user can
exclude a forward-compatible set of things that should be recomputable
given existing data.
V2 changes:
* Updated the documentation for core.fsyncmethod to be less certain.
writeout-only probably does not do the right thing on Linux.
* Split out the core.fsync=index change into its own commit.
* Rename REPO_COMPONENT to FSYNC_COMPONENT. This is really specific to
fsyncing, so the name should reflect that.
* Re-add missing Makefile change for SYNC_FILE_RANGE.
* Tested writeout-only mode, index syncing, and general config settings.
[1] https://lore.kernel.org/git/211110.86r1bogg27.gmgdl@evledraar.gmail.com/
[2]
https://lore.kernel.org/git/dd65718814011eb93ccc4428f9882e0f025224a6.1636029491.git.ps@pks.im/
[3]
https://lore.kernel.org/git/pull.1076.git.git.1629856292.gitgitgadget@gmail.com/
[4]
https://lore.kernel.org/git/CANQDOdf8C4-haK9=Q_J4Cid8bQALnmGDm=SvatRbaVf+tkzqLw@mail.gmail.com/
[5] https://lore.kernel.org/git/211207.861r2opplg.gmgdl@evledraar.gmail.com/
Neeraj Singh (4):
core.fsyncmethod: add writeout-only mode
core.fsync: introduce granular fsync control
core.fsync: new option to harden the index
core.fsync: add a `derived-metadata` aggregate option
Documentation/config/core.txt | 35 ++++++++---
Makefile | 6 ++
builtin/fast-import.c | 2 +-
builtin/index-pack.c | 4 +-
builtin/pack-objects.c | 8 ++-
bulk-checkin.c | 5 +-
cache.h | 49 +++++++++++++++-
commit-graph.c | 3 +-
compat/mingw.h | 3 +
compat/win32/flush.c | 28 +++++++++
config.c | 90 ++++++++++++++++++++++++++++-
config.mak.uname | 3 +
configure.ac | 8 +++
contrib/buildsystems/CMakeLists.txt | 3 +-
csum-file.c | 5 +-
csum-file.h | 3 +-
environment.c | 3 +-
git-compat-util.h | 24 ++++++++
midx.c | 3 +-
object-file.c | 3 +-
pack-bitmap-write.c | 3 +-
pack-write.c | 13 +++--
read-cache.c | 19 ++++--
wrapper.c | 64 ++++++++++++++++++++
write-or-die.c | 10 ++--
25 files changed, 354 insertions(+), 43 deletions(-)
create mode 100644 compat/win32/flush.c
base-commit: abe6bb3905392d5eb6b01fa6e54d7e784e0522aa
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1093%2Fneerajsi-msft%2Fns%2Fcore-fsync-v3
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1093/neerajsi-msft/ns/core-fsync-v3
Pull-Request: https://github.com/gitgitgadget/git/pull/1093
Range-diff vs v2:
1: e79522cbdd4 ! 1: 15edfe51509 core.fsyncmethod: add writeout-only mode
@@ Metadata
## Commit message ##
core.fsyncmethod: add writeout-only mode
- This commit introduces the `core.fsyncmethod` configuration
+ This commit introduces the `core.fsyncMethod` configuration
knob, which can currently be set to `fsync` or `writeout-only`.
The new writeout-only mode attempts to tell the operating system to
@@ Commit message
directive to the storage controller. This change updates fsync to do
fcntl(F_FULLFSYNC) to make fsync actually durable. We maintain parity
with existing behavior on Apple platforms by setting the default value
- of the new core.fsyncmethod option.
+ of the new core.fsyncMethod option.
Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
@@ compat/mingw.h: int mingw_getpagesize(void);
## compat/win32/flush.c (new) ##
@@
-+#include "../../git-compat-util.h"
++#include "git-compat-util.h"
+#include <winternl.h>
+#include "lazyload.h"
+
@@ config.c: static int git_default_core_config(const char *var, const char *value,
+ else if (!strcmp(value, "writeout-only"))
+ fsync_method = FSYNC_METHOD_WRITEOUT_ONLY;
+ else
-+ warning(_("unknown %s value '%s'"), var, value);
++ warning(_("ignoring unknown core.fsyncMethod value '%s'"), value);
+
+ }
+
@@ git-compat-util.h: __attribute__((format (printf, 1, 2))) NORETURN
+#endif
+
+enum fsync_action {
-+ FSYNC_WRITEOUT_ONLY,
-+ FSYNC_HARDWARE_FLUSH
++ FSYNC_WRITEOUT_ONLY,
++ FSYNC_HARDWARE_FLUSH
+};
+
+/*
@@ wrapper.c: int xmkstemp_mode(char *filename_template, int mode)
return fd;
}
++/*
++ * Some platforms return EINTR from fsync. Since fsync is invoked in some
++ * cases by a wrapper that dies on failure, do not expose EINTR to callers.
++ */
++static int fsync_loop(int fd)
++{
++ int err;
++
++ do {
++ err = fsync(fd);
++ } while (err < 0 && errno == EINTR);
++ return err;
++}
++
+int git_fsync(int fd, enum fsync_action action)
+{
+ switch (action) {
@@ wrapper.c: int xmkstemp_mode(char *filename_template, int mode)
+ * on macOS, fsync just causes filesystem cache writeback but does not
+ * flush hardware caches.
+ */
-+ return fsync(fd);
++ return fsync_loop(fd);
+#endif
+
+#ifdef HAVE_SYNC_FILE_RANGE
@@ wrapper.c: int xmkstemp_mode(char *filename_template, int mode)
+ * case, since callers asking for a hardware flush may die if
+ * this function returns an error.
+ */
-+ for (;;) {
-+ int err;
+#ifdef __APPLE__
-+ err = fcntl(fd, F_FULLFSYNC);
++ return fcntl(fd, F_FULLFSYNC);
+#else
-+ err = fsync(fd);
++ return fsync_loop(fd);
+#endif
-+ if (err >= 0 || errno != EINTR)
-+ return err;
-+ }
-+
+ default:
+ BUG("unexpected git_fsync(%d) call", action);
+ }
2: ff80a94bf9a ! 2: 080be1a6f64 core.fsync: introduce granular fsync control
@@ config.c: static int git_parse_maybe_bool_text(const char *value)
+
+ if (!found) {
+ char *component = xstrndup(string, len);
-+ warning(_("unknown %s value '%s'"), var, component);
++ warning(_("ignoring unknown core.fsync component '%s'"), component);
+ free(component);
+ }
+
3: 86e39b8f8d1 = 3: 2207950beba core.fsync: new option to harden the index
-: ----------- > 4: a830d177d4c core.fsync: add a `derived-metadata` aggregate option
--
gitgitgadget
^ permalink raw reply [flat|nested] 122+ messages in thread
* [PATCH v3 1/4] core.fsyncmethod: add writeout-only mode
2021-12-09 0:57 ` [PATCH v3 0/4] " Neeraj K. Singh via GitGitGadget
@ 2021-12-09 0:57 ` Neeraj Singh via GitGitGadget
2021-12-09 0:57 ` [PATCH v3 2/4] core.fsync: introduce granular fsync control Neeraj Singh via GitGitGadget
` (4 subsequent siblings)
5 siblings, 0 replies; 122+ messages in thread
From: Neeraj Singh via GitGitGadget @ 2021-12-09 0:57 UTC (permalink / raw)
To: git
Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps,
Neeraj K. Singh, Neeraj Singh
From: Neeraj Singh <neerajsi@microsoft.com>
This commit introduces the `core.fsyncMethod` configuration
knob, which can currently be set to `fsync` or `writeout-only`.
The new writeout-only mode attempts to tell the operating system to
flush its in-memory page cache to the storage hardware without issuing a
CACHE_FLUSH command to the storage controller.
Writeout-only fsync is significantly faster than a vanilla fsync on
common hardware, since data is written to a disk-side cache rather than
all the way to a durable medium. Later changes in this patch series will
take advantage of this primitive to implement batching of hardware
flushes.
When git_fsync is called with FSYNC_WRITEOUT_ONLY, it may fail and the
caller is expected to do an ordinary fsync as needed.
On Apple platforms, the fsync system call does not issue a CACHE_FLUSH
directive to the storage controller. This change updates fsync to do
fcntl(F_FULLFSYNC) to make fsync actually durable. We maintain parity
with existing behavior on Apple platforms by setting the default value
of the new core.fsyncMethod option.
Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
---
Documentation/config/core.txt | 9 ++++
Makefile | 6 +++
cache.h | 7 ++++
compat/mingw.h | 3 ++
compat/win32/flush.c | 28 +++++++++++++
config.c | 12 ++++++
config.mak.uname | 3 ++
configure.ac | 8 ++++
contrib/buildsystems/CMakeLists.txt | 3 +-
environment.c | 2 +-
git-compat-util.h | 24 +++++++++++
wrapper.c | 64 +++++++++++++++++++++++++++++
write-or-die.c | 10 +++--
13 files changed, 173 insertions(+), 6 deletions(-)
create mode 100644 compat/win32/flush.c
diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
index c04f62a54a1..dbb134f7136 100644
--- a/Documentation/config/core.txt
+++ b/Documentation/config/core.txt
@@ -547,6 +547,15 @@ core.whitespace::
is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent`
errors. The default tab width is 8. Allowed values are 1 to 63.
+core.fsyncMethod::
+ A value indicating the strategy Git will use to harden repository data
+ using fsync and related primitives.
++
+* `fsync` uses the fsync() system call or platform equivalents.
+* `writeout-only` issues pagecache writeback requests, but depending on the
+ filesystem and storage hardware, data added to the repository may not be
+ durable in the event of a system crash. This is the default mode on macOS.
+
core.fsyncObjectFiles::
This boolean will enable 'fsync()' when writing object files.
+
diff --git a/Makefile b/Makefile
index d56c0e4aadc..cba024615c9 100644
--- a/Makefile
+++ b/Makefile
@@ -403,6 +403,8 @@ all::
#
# Define HAVE_CLOCK_MONOTONIC if your platform has CLOCK_MONOTONIC.
#
+# Define HAVE_SYNC_FILE_RANGE if your platform has sync_file_range.
+#
# Define NEEDS_LIBRT if your platform requires linking with librt (glibc version
# before 2.17) for clock_gettime and CLOCK_MONOTONIC.
#
@@ -1881,6 +1883,10 @@ ifdef HAVE_CLOCK_MONOTONIC
BASIC_CFLAGS += -DHAVE_CLOCK_MONOTONIC
endif
+ifdef HAVE_SYNC_FILE_RANGE
+ BASIC_CFLAGS += -DHAVE_SYNC_FILE_RANGE
+endif
+
ifdef NEEDS_LIBRT
EXTLIBS += -lrt
endif
diff --git a/cache.h b/cache.h
index eba12487b99..9cd60d94952 100644
--- a/cache.h
+++ b/cache.h
@@ -986,6 +986,13 @@ extern int read_replace_refs;
extern char *git_replace_ref_base;
extern int fsync_object_files;
+
+enum fsync_method {
+ FSYNC_METHOD_FSYNC,
+ FSYNC_METHOD_WRITEOUT_ONLY
+};
+
+extern enum fsync_method fsync_method;
extern int core_preload_index;
extern int precomposed_unicode;
extern int protect_hfs;
diff --git a/compat/mingw.h b/compat/mingw.h
index c9a52ad64a6..6074a3d3ced 100644
--- a/compat/mingw.h
+++ b/compat/mingw.h
@@ -329,6 +329,9 @@ int mingw_getpagesize(void);
#define getpagesize mingw_getpagesize
#endif
+int win32_fsync_no_flush(int fd);
+#define fsync_no_flush win32_fsync_no_flush
+
struct rlimit {
unsigned int rlim_cur;
};
diff --git a/compat/win32/flush.c b/compat/win32/flush.c
new file mode 100644
index 00000000000..6ca82f1dae5
--- /dev/null
+++ b/compat/win32/flush.c
@@ -0,0 +1,28 @@
+#include "git-compat-util.h"
+#include <winternl.h>
+#include "lazyload.h"
+
+int win32_fsync_no_flush(int fd)
+{
+ IO_STATUS_BLOCK io_status;
+
+#define FLUSH_FLAGS_FILE_DATA_ONLY 1
+
+ DECLARE_PROC_ADDR(ntdll.dll, NTSTATUS, NtFlushBuffersFileEx,
+ HANDLE FileHandle, ULONG Flags, PVOID Parameters, ULONG ParameterSize,
+ PIO_STATUS_BLOCK IoStatusBlock);
+
+ if (!INIT_PROC_ADDR(NtFlushBuffersFileEx)) {
+ errno = ENOSYS;
+ return -1;
+ }
+
+ memset(&io_status, 0, sizeof(io_status));
+ if (NtFlushBuffersFileEx((HANDLE)_get_osfhandle(fd), FLUSH_FLAGS_FILE_DATA_ONLY,
+ NULL, 0, &io_status)) {
+ errno = EINVAL;
+ return -1;
+ }
+
+ return 0;
+}
diff --git a/config.c b/config.c
index c5873f3a706..139df71ba17 100644
--- a/config.c
+++ b/config.c
@@ -1490,6 +1490,18 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
return 0;
}
+ if (!strcmp(var, "core.fsyncmethod")) {
+ if (!value)
+ return config_error_nonbool(var);
+ if (!strcmp(value, "fsync"))
+ fsync_method = FSYNC_METHOD_FSYNC;
+ else if (!strcmp(value, "writeout-only"))
+ fsync_method = FSYNC_METHOD_WRITEOUT_ONLY;
+ else
+ warning(_("ignoring unknown core.fsyncMethod value '%s'"), value);
+
+ }
+
if (!strcmp(var, "core.fsyncobjectfiles")) {
fsync_object_files = git_config_bool(var, value);
return 0;
diff --git a/config.mak.uname b/config.mak.uname
index d0701f9beb0..774a09622d2 100644
--- a/config.mak.uname
+++ b/config.mak.uname
@@ -57,6 +57,7 @@ ifeq ($(uname_S),Linux)
HAVE_CLOCK_MONOTONIC = YesPlease
# -lrt is needed for clock_gettime on glibc <= 2.16
NEEDS_LIBRT = YesPlease
+ HAVE_SYNC_FILE_RANGE = YesPlease
HAVE_GETDELIM = YesPlease
FREAD_READS_DIRECTORIES = UnfortunatelyYes
BASIC_CFLAGS += -DHAVE_SYSINFO
@@ -453,6 +454,7 @@ endif
CFLAGS =
BASIC_CFLAGS = -nologo -I. -Icompat/vcbuild/include -DWIN32 -D_CONSOLE -DHAVE_STRING_H -D_CRT_SECURE_NO_WARNINGS -D_CRT_NONSTDC_NO_DEPRECATE
COMPAT_OBJS = compat/msvc.o compat/winansi.o \
+ compat/win32/flush.o \
compat/win32/path-utils.o \
compat/win32/pthread.o compat/win32/syslog.o \
compat/win32/trace2_win32_process_info.o \
@@ -628,6 +630,7 @@ ifeq ($(uname_S),MINGW)
COMPAT_CFLAGS += -DSTRIP_EXTENSION=\".exe\"
COMPAT_OBJS += compat/mingw.o compat/winansi.o \
compat/win32/trace2_win32_process_info.o \
+ compat/win32/flush.o \
compat/win32/path-utils.o \
compat/win32/pthread.o compat/win32/syslog.o \
compat/win32/dirent.o
diff --git a/configure.ac b/configure.ac
index 5ee25ec95c8..6bd6bef1c44 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1082,6 +1082,14 @@ AC_COMPILE_IFELSE([CLOCK_MONOTONIC_SRC],
[AC_MSG_RESULT([no])
HAVE_CLOCK_MONOTONIC=])
GIT_CONF_SUBST([HAVE_CLOCK_MONOTONIC])
+
+#
+# Define HAVE_SYNC_FILE_RANGE=YesPlease if sync_file_range is available.
+GIT_CHECK_FUNC(sync_file_range,
+ [HAVE_SYNC_FILE_RANGE=YesPlease],
+ [HAVE_SYNC_FILE_RANGE])
+GIT_CONF_SUBST([HAVE_SYNC_FILE_RANGE])
+
#
# Define NO_SETITIMER if you don't have setitimer.
GIT_CHECK_FUNC(setitimer,
diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt
index 86b46114464..6d7bc16d054 100644
--- a/contrib/buildsystems/CMakeLists.txt
+++ b/contrib/buildsystems/CMakeLists.txt
@@ -261,7 +261,8 @@ if(CMAKE_SYSTEM_NAME STREQUAL "Windows")
NOGDI OBJECT_CREATION_MODE=1 __USE_MINGW_ANSI_STDIO=0
USE_NED_ALLOCATOR OVERRIDE_STRDUP MMAP_PREVENTS_DELETE USE_WIN32_MMAP
UNICODE _UNICODE HAVE_WPGMPTR ENSURE_MSYSTEM_IS_SET)
- list(APPEND compat_SOURCES compat/mingw.c compat/winansi.c compat/win32/path-utils.c
+ list(APPEND compat_SOURCES compat/mingw.c compat/winansi.c
+ compat/win32/flush.c compat/win32/path-utils.c
compat/win32/pthread.c compat/win32mmap.c compat/win32/syslog.c
compat/win32/trace2_win32_process_info.c compat/win32/dirent.c
compat/nedmalloc/nedmalloc.c compat/strdup.c)
diff --git a/environment.c b/environment.c
index 9da7f3c1a19..f9140e842cf 100644
--- a/environment.c
+++ b/environment.c
@@ -41,7 +41,7 @@ const char *git_attributes_file;
const char *git_hooks_path;
int zlib_compression_level = Z_BEST_SPEED;
int pack_compression_level = Z_DEFAULT_COMPRESSION;
-int fsync_object_files;
+enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT;
size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE;
size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT;
size_t delta_base_cache_limit = 96 * 1024 * 1024;
diff --git a/git-compat-util.h b/git-compat-util.h
index c6bd2a84e55..50db85a8610 100644
--- a/git-compat-util.h
+++ b/git-compat-util.h
@@ -1239,6 +1239,30 @@ __attribute__((format (printf, 1, 2))) NORETURN
void BUG(const char *fmt, ...);
#endif
+#ifdef __APPLE__
+#define FSYNC_METHOD_DEFAULT FSYNC_METHOD_WRITEOUT_ONLY
+#else
+#define FSYNC_METHOD_DEFAULT FSYNC_METHOD_FSYNC
+#endif
+
+enum fsync_action {
+ FSYNC_WRITEOUT_ONLY,
+ FSYNC_HARDWARE_FLUSH
+};
+
+/*
+ * Issues an fsync against the specified file according to the specified mode.
+ *
+ * FSYNC_WRITEOUT_ONLY attempts to use interfaces available on some operating
+ * systems to flush the OS cache without issuing a flush command to the storage
+ * controller. If those interfaces are unavailable, the function fails with
+ * ENOSYS.
+ *
+ * FSYNC_HARDWARE_FLUSH does an OS writeout and hardware flush to ensure that
+ * changes are durable. It is not expected to fail.
+ */
+int git_fsync(int fd, enum fsync_action action);
+
/*
* Preserves errno, prints a message, but gives no warning for ENOENT.
* Returns 0 on success, which includes trying to unlink an object that does
diff --git a/wrapper.c b/wrapper.c
index 36e12119d76..572f28f14ff 100644
--- a/wrapper.c
+++ b/wrapper.c
@@ -546,6 +546,70 @@ int xmkstemp_mode(char *filename_template, int mode)
return fd;
}
+/*
+ * Some platforms return EINTR from fsync. Since fsync is invoked in some
+ * cases by a wrapper that dies on failure, do not expose EINTR to callers.
+ */
+static int fsync_loop(int fd)
+{
+ int err;
+
+ do {
+ err = fsync(fd);
+ } while (err < 0 && errno == EINTR);
+ return err;
+}
+
+int git_fsync(int fd, enum fsync_action action)
+{
+ switch (action) {
+ case FSYNC_WRITEOUT_ONLY:
+
+#ifdef __APPLE__
+ /*
+ * on macOS, fsync just causes filesystem cache writeback but does not
+ * flush hardware caches.
+ */
+ return fsync_loop(fd);
+#endif
+
+#ifdef HAVE_SYNC_FILE_RANGE
+ /*
+ * On linux 2.6.17 and above, sync_file_range is the way to issue
+ * a writeback without a hardware flush. An offset of 0 and size of 0
+ * indicates writeout of the entire file and the wait flags ensure that all
+ * dirty data is written to the disk (potentially in a disk-side cache)
+ * before we continue.
+ */
+
+ return sync_file_range(fd, 0, 0, SYNC_FILE_RANGE_WAIT_BEFORE |
+ SYNC_FILE_RANGE_WRITE |
+ SYNC_FILE_RANGE_WAIT_AFTER);
+#endif
+
+#ifdef fsync_no_flush
+ return fsync_no_flush(fd);
+#endif
+
+ errno = ENOSYS;
+ return -1;
+
+ case FSYNC_HARDWARE_FLUSH:
+ /*
+ * On some platforms fsync may return EINTR. Try again in this
+ * case, since callers asking for a hardware flush may die if
+ * this function returns an error.
+ */
+#ifdef __APPLE__
+ return fcntl(fd, F_FULLFSYNC);
+#else
+ return fsync_loop(fd);
+#endif
+ default:
+ BUG("unexpected git_fsync(%d) call", action);
+ }
+}
+
static int warn_if_unremovable(const char *op, const char *file, int rc)
{
int err;
diff --git a/write-or-die.c b/write-or-die.c
index 0b1ec8190b6..0702acdd5e8 100644
--- a/write-or-die.c
+++ b/write-or-die.c
@@ -57,10 +57,12 @@ void fprintf_or_die(FILE *f, const char *fmt, ...)
void fsync_or_die(int fd, const char *msg)
{
- while (fsync(fd) < 0) {
- if (errno != EINTR)
- die_errno("fsync error on '%s'", msg);
- }
+ if (fsync_method == FSYNC_METHOD_WRITEOUT_ONLY &&
+ git_fsync(fd, FSYNC_WRITEOUT_ONLY) >= 0)
+ return;
+
+ if (git_fsync(fd, FSYNC_HARDWARE_FLUSH) < 0)
+ die_errno("fsync error on '%s'", msg);
}
void write_or_die(int fd, const void *buf, size_t count)
--
gitgitgadget
^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v3 2/4] core.fsync: introduce granular fsync control
2021-12-09 0:57 ` [PATCH v3 0/4] " Neeraj K. Singh via GitGitGadget
2021-12-09 0:57 ` [PATCH v3 1/4] core.fsyncmethod: add writeout-only mode Neeraj Singh via GitGitGadget
@ 2021-12-09 0:57 ` Neeraj Singh via GitGitGadget
2021-12-09 0:57 ` [PATCH v3 3/4] core.fsync: new option to harden the index Neeraj Singh via GitGitGadget
` (3 subsequent siblings)
5 siblings, 0 replies; 122+ messages in thread
From: Neeraj Singh via GitGitGadget @ 2021-12-09 0:57 UTC (permalink / raw)
To: git
Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps,
Neeraj K. Singh, Neeraj Singh
From: Neeraj Singh <neerajsi@microsoft.com>
This commit introduces the `core.fsync` configuration
knob which can be used to control how components of the
repository are made durable on disk.
This setting allows future extensibility of the list of
syncable components:
* We issue a warning rather than an error for unrecognized
components, so new configs can be used with old Git versions.
* We support negation, so users can choose one of the default
aggregate options and then remove components that they don't
want. The user would then harden any new components added in
a Git version update.
This also supports the common request of doing absolutely no
fysncing with the `core.fsync=none` value, which is expected
to make the test suite faster.
Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
---
Documentation/config/core.txt | 27 +++++++++----
builtin/fast-import.c | 2 +-
builtin/index-pack.c | 4 +-
builtin/pack-objects.c | 8 ++--
bulk-checkin.c | 5 ++-
cache.h | 39 +++++++++++++++++-
commit-graph.c | 3 +-
config.c | 76 ++++++++++++++++++++++++++++++++++-
csum-file.c | 5 ++-
csum-file.h | 3 +-
environment.c | 1 +
midx.c | 3 +-
object-file.c | 3 +-
pack-bitmap-write.c | 3 +-
pack-write.c | 13 +++---
read-cache.c | 2 +-
16 files changed, 164 insertions(+), 33 deletions(-)
diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
index dbb134f7136..4f1747ec871 100644
--- a/Documentation/config/core.txt
+++ b/Documentation/config/core.txt
@@ -547,6 +547,25 @@ core.whitespace::
is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent`
errors. The default tab width is 8. Allowed values are 1 to 63.
+core.fsync::
+ A comma-separated list of parts of the repository which should be
+ hardened via the core.fsyncMethod when created or modified. You can
+ disable hardening of any component by prefixing it with a '-'. Later
+ items take precedence over earlier ones in the list. For example,
+ `core.fsync=all,-pack-metadata` means "harden everything except pack
+ metadata." Items that are not hardened may be lost in the event of an
+ unclean system shutdown.
++
+* `none` disables fsync completely. This must be specified alone.
+* `loose-object` hardens objects added to the repo in loose-object form.
+* `pack` hardens objects added to the repo in packfile form.
+* `pack-metadata` hardens packfile bitmaps and indexes.
+* `commit-graph` hardens the commit graph file.
+* `objects` is an aggregate option that includes `loose-objects`, `pack`,
+ `pack-metadata`, and `commit-graph`.
+* `default` is an aggregate option that is equivalent to `objects,-loose-object`
+* `all` is an aggregate option that syncs all individual components above.
+
core.fsyncMethod::
A value indicating the strategy Git will use to harden repository data
using fsync and related primitives.
@@ -556,14 +575,6 @@ core.fsyncMethod::
filesystem and storage hardware, data added to the repository may not be
durable in the event of a system crash. This is the default mode on macOS.
-core.fsyncObjectFiles::
- This boolean will enable 'fsync()' when writing object files.
-+
-This is a total waste of time and effort on a filesystem that orders
-data writes properly, but can be useful for filesystems that do not use
-journalling (traditional UNIX filesystems) or that only journal metadata
-and not file contents (OS X's HFS+, or Linux ext3 with "data=writeback").
-
core.preloadIndex::
Enable parallel index preload for operations like 'git diff'
+
diff --git a/builtin/fast-import.c b/builtin/fast-import.c
index 20406f67754..e27a4580f85 100644
--- a/builtin/fast-import.c
+++ b/builtin/fast-import.c
@@ -856,7 +856,7 @@ static void end_packfile(void)
struct tag *t;
close_pack_windows(pack_data);
- finalize_hashfile(pack_file, cur_pack_oid.hash, 0);
+ finalize_hashfile(pack_file, cur_pack_oid.hash, FSYNC_COMPONENT_PACK, 0);
fixup_pack_header_footer(pack_data->pack_fd, pack_data->hash,
pack_data->pack_name, object_count,
cur_pack_oid.hash, pack_size);
diff --git a/builtin/index-pack.c b/builtin/index-pack.c
index c23d01de7dc..c32534c13b4 100644
--- a/builtin/index-pack.c
+++ b/builtin/index-pack.c
@@ -1286,7 +1286,7 @@ static void conclude_pack(int fix_thin_pack, const char *curr_pack, unsigned cha
nr_objects - nr_objects_initial);
stop_progress_msg(&progress, msg.buf);
strbuf_release(&msg);
- finalize_hashfile(f, tail_hash, 0);
+ finalize_hashfile(f, tail_hash, FSYNC_COMPONENT_PACK, 0);
hashcpy(read_hash, pack_hash);
fixup_pack_header_footer(output_fd, pack_hash,
curr_pack, nr_objects,
@@ -1508,7 +1508,7 @@ static void final(const char *final_pack_name, const char *curr_pack_name,
if (!from_stdin) {
close(input_fd);
} else {
- fsync_or_die(output_fd, curr_pack_name);
+ fsync_component_or_die(FSYNC_COMPONENT_PACK, output_fd, curr_pack_name);
err = close(output_fd);
if (err)
die_errno(_("error while closing pack file"));
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index 857be7826f3..916c55d6ce9 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -1204,11 +1204,13 @@ static void write_pack_file(void)
* If so, rewrite it like in fast-import
*/
if (pack_to_stdout) {
- finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_CLOSE);
+ finalize_hashfile(f, hash, FSYNC_COMPONENT_NONE,
+ CSUM_HASH_IN_STREAM | CSUM_CLOSE);
} else if (nr_written == nr_remaining) {
- finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
+ finalize_hashfile(f, hash, FSYNC_COMPONENT_PACK,
+ CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
} else {
- int fd = finalize_hashfile(f, hash, 0);
+ int fd = finalize_hashfile(f, hash, FSYNC_COMPONENT_PACK, 0);
fixup_pack_header_footer(fd, hash, pack_tmp_name,
nr_written, hash, offset);
close(fd);
diff --git a/bulk-checkin.c b/bulk-checkin.c
index 8785b2ac806..a2cf9dcbc8d 100644
--- a/bulk-checkin.c
+++ b/bulk-checkin.c
@@ -53,9 +53,10 @@ static void finish_bulk_checkin(struct bulk_checkin_state *state)
unlink(state->pack_tmp_name);
goto clear_exit;
} else if (state->nr_written == 1) {
- finalize_hashfile(state->f, hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
+ finalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK,
+ CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
} else {
- int fd = finalize_hashfile(state->f, hash, 0);
+ int fd = finalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK, 0);
fixup_pack_header_footer(fd, hash, state->pack_tmp_name,
state->nr_written, hash,
state->offset);
diff --git a/cache.h b/cache.h
index 9cd60d94952..d83fbaf2619 100644
--- a/cache.h
+++ b/cache.h
@@ -985,7 +985,38 @@ void reset_shared_repository(void);
extern int read_replace_refs;
extern char *git_replace_ref_base;
-extern int fsync_object_files;
+/*
+ * These values are used to help identify parts of a repository to fsync.
+ * FSYNC_COMPONENT_NONE identifies data that will not be a persistent part of the
+ * repository and so shouldn't be fsynced.
+ */
+enum fsync_component {
+ FSYNC_COMPONENT_NONE = 0,
+ FSYNC_COMPONENT_LOOSE_OBJECT = 1 << 0,
+ FSYNC_COMPONENT_PACK = 1 << 1,
+ FSYNC_COMPONENT_PACK_METADATA = 1 << 2,
+ FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3,
+};
+
+#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \
+ FSYNC_COMPONENT_PACK_METADATA | \
+ FSYNC_COMPONENT_COMMIT_GRAPH)
+
+#define FSYNC_COMPONENTS_OBJECTS (FSYNC_COMPONENT_LOOSE_OBJECT | \
+ FSYNC_COMPONENT_PACK | \
+ FSYNC_COMPONENT_PACK_METADATA | \
+ FSYNC_COMPONENT_COMMIT_GRAPH)
+
+#define FSYNC_COMPONENTS_ALL (FSYNC_COMPONENT_LOOSE_OBJECT | \
+ FSYNC_COMPONENT_PACK | \
+ FSYNC_COMPONENT_PACK_METADATA | \
+ FSYNC_COMPONENT_COMMIT_GRAPH)
+
+
+/*
+ * A bitmask indicating which components of the repo should be fsynced.
+ */
+extern enum fsync_component fsync_components;
enum fsync_method {
FSYNC_METHOD_FSYNC,
@@ -1747,6 +1778,12 @@ int copy_file_with_time(const char *dst, const char *src, int mode);
void write_or_die(int fd, const void *buf, size_t count);
void fsync_or_die(int fd, const char *);
+inline void fsync_component_or_die(enum fsync_component component, int fd, const char *msg)
+{
+ if (fsync_components & component)
+ fsync_or_die(fd, msg);
+}
+
ssize_t read_in_full(int fd, void *buf, size_t count);
ssize_t write_in_full(int fd, const void *buf, size_t count);
ssize_t pread_in_full(int fd, void *buf, size_t count, off_t offset);
diff --git a/commit-graph.c b/commit-graph.c
index 2706683acfe..c8a5dea4541 100644
--- a/commit-graph.c
+++ b/commit-graph.c
@@ -1939,7 +1939,8 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)
}
close_commit_graph(ctx->r->objects);
- finalize_hashfile(f, file_hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC);
+ finalize_hashfile(f, file_hash, FSYNC_COMPONENT_COMMIT_GRAPH,
+ CSUM_HASH_IN_STREAM | CSUM_FSYNC);
free_chunkfile(cf);
if (ctx->split) {
diff --git a/config.c b/config.c
index 139df71ba17..5ab381388f9 100644
--- a/config.c
+++ b/config.c
@@ -1213,6 +1213,73 @@ static int git_parse_maybe_bool_text(const char *value)
return -1;
}
+static const struct fsync_component_entry {
+ const char *name;
+ enum fsync_component component_bits;
+} fsync_component_table[] = {
+ { "loose-object", FSYNC_COMPONENT_LOOSE_OBJECT },
+ { "pack", FSYNC_COMPONENT_PACK },
+ { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA },
+ { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH },
+ { "objects", FSYNC_COMPONENTS_OBJECTS },
+ { "default", FSYNC_COMPONENTS_DEFAULT },
+ { "all", FSYNC_COMPONENTS_ALL },
+};
+
+static enum fsync_component parse_fsync_components(const char *var, const char *string)
+{
+ enum fsync_component output = 0;
+
+ if (!strcmp(string, "none"))
+ return output;
+
+ while (string) {
+ int i;
+ size_t len;
+ const char *ep;
+ int negated = 0;
+ int found = 0;
+
+ string = string + strspn(string, ", \t\n\r");
+ ep = strchrnul(string, ',');
+ len = ep - string;
+
+ if (*string == '-') {
+ negated = 1;
+ string++;
+ len--;
+ if (!len)
+ warning(_("invalid value for variable %s"), var);
+ }
+
+ if (!len)
+ break;
+
+ for (i = 0; i < ARRAY_SIZE(fsync_component_table); ++i) {
+ const struct fsync_component_entry *entry = &fsync_component_table[i];
+
+ if (strncmp(entry->name, string, len))
+ continue;
+
+ found = 1;
+ if (negated)
+ output &= ~entry->component_bits;
+ else
+ output |= entry->component_bits;
+ }
+
+ if (!found) {
+ char *component = xstrndup(string, len);
+ warning(_("ignoring unknown core.fsync component '%s'"), component);
+ free(component);
+ }
+
+ string = ep;
+ }
+
+ return output;
+}
+
int git_parse_maybe_bool(const char *value)
{
int v = git_parse_maybe_bool_text(value);
@@ -1490,6 +1557,13 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
return 0;
}
+ if (!strcmp(var, "core.fsync")) {
+ if (!value)
+ return config_error_nonbool(var);
+ fsync_components = parse_fsync_components(var, value);
+ return 0;
+ }
+
if (!strcmp(var, "core.fsyncmethod")) {
if (!value)
return config_error_nonbool(var);
@@ -1503,7 +1577,7 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
}
if (!strcmp(var, "core.fsyncobjectfiles")) {
- fsync_object_files = git_config_bool(var, value);
+ warning(_("core.fsyncobjectfiles is deprecated; use core.fsync instead"));
return 0;
}
diff --git a/csum-file.c b/csum-file.c
index 26e8a6df44e..59ef3398ca2 100644
--- a/csum-file.c
+++ b/csum-file.c
@@ -58,7 +58,8 @@ static void free_hashfile(struct hashfile *f)
free(f);
}
-int finalize_hashfile(struct hashfile *f, unsigned char *result, unsigned int flags)
+int finalize_hashfile(struct hashfile *f, unsigned char *result,
+ enum fsync_component component, unsigned int flags)
{
int fd;
@@ -69,7 +70,7 @@ int finalize_hashfile(struct hashfile *f, unsigned char *result, unsigned int fl
if (flags & CSUM_HASH_IN_STREAM)
flush(f, f->buffer, the_hash_algo->rawsz);
if (flags & CSUM_FSYNC)
- fsync_or_die(f->fd, f->name);
+ fsync_component_or_die(component, f->fd, f->name);
if (flags & CSUM_CLOSE) {
if (close(f->fd))
die_errno("%s: sha1 file error on close", f->name);
diff --git a/csum-file.h b/csum-file.h
index 291215b34eb..0d29f528fbc 100644
--- a/csum-file.h
+++ b/csum-file.h
@@ -1,6 +1,7 @@
#ifndef CSUM_FILE_H
#define CSUM_FILE_H
+#include "cache.h"
#include "hash.h"
struct progress;
@@ -38,7 +39,7 @@ int hashfile_truncate(struct hashfile *, struct hashfile_checkpoint *);
struct hashfile *hashfd(int fd, const char *name);
struct hashfile *hashfd_check(const char *name);
struct hashfile *hashfd_throughput(int fd, const char *name, struct progress *tp);
-int finalize_hashfile(struct hashfile *, unsigned char *, unsigned int);
+int finalize_hashfile(struct hashfile *, unsigned char *, enum fsync_component, unsigned int);
void hashwrite(struct hashfile *, const void *, unsigned int);
void hashflush(struct hashfile *f);
void crc32_begin(struct hashfile *);
diff --git a/environment.c b/environment.c
index f9140e842cf..09905adecf9 100644
--- a/environment.c
+++ b/environment.c
@@ -42,6 +42,7 @@ const char *git_hooks_path;
int zlib_compression_level = Z_BEST_SPEED;
int pack_compression_level = Z_DEFAULT_COMPRESSION;
enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT;
+enum fsync_component fsync_components = FSYNC_COMPONENTS_DEFAULT;
size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE;
size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT;
size_t delta_base_cache_limit = 96 * 1024 * 1024;
diff --git a/midx.c b/midx.c
index 837b46b2af5..882f91f7d57 100644
--- a/midx.c
+++ b/midx.c
@@ -1406,7 +1406,8 @@ static int write_midx_internal(const char *object_dir,
write_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs);
write_chunkfile(cf, &ctx);
- finalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);
+ finalize_hashfile(f, midx_hash, FSYNC_COMPONENT_PACK_METADATA,
+ CSUM_FSYNC | CSUM_HASH_IN_STREAM);
free_chunkfile(cf);
if (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP))
diff --git a/object-file.c b/object-file.c
index eb972cdccd2..9d9c4a39e85 100644
--- a/object-file.c
+++ b/object-file.c
@@ -1809,8 +1809,7 @@ int hash_object_file(const struct git_hash_algo *algo, const void *buf,
/* Finalize a file on disk, and close it. */
static void close_loose_object(int fd)
{
- if (fsync_object_files)
- fsync_or_die(fd, "loose object file");
+ fsync_component_or_die(FSYNC_COMPONENT_LOOSE_OBJECT, fd, "loose object file");
if (close(fd) != 0)
die_errno(_("error when closing loose object file"));
}
diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c
index 9c55c1531e1..c16e43d1669 100644
--- a/pack-bitmap-write.c
+++ b/pack-bitmap-write.c
@@ -719,7 +719,8 @@ void bitmap_writer_finish(struct pack_idx_entry **index,
if (options & BITMAP_OPT_HASH_CACHE)
write_hash_cache(f, index, index_nr);
- finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
+ finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA,
+ CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
if (adjust_shared_perm(tmp_file.buf))
die_errno("unable to make temporary bitmap file readable");
diff --git a/pack-write.c b/pack-write.c
index a5846f3a346..51812cb1299 100644
--- a/pack-write.c
+++ b/pack-write.c
@@ -159,9 +159,9 @@ const char *write_idx_file(const char *index_name, struct pack_idx_entry **objec
}
hashwrite(f, sha1, the_hash_algo->rawsz);
- finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE |
- ((opts->flags & WRITE_IDX_VERIFY)
- ? 0 : CSUM_FSYNC));
+ finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA,
+ CSUM_HASH_IN_STREAM | CSUM_CLOSE |
+ ((opts->flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC));
return index_name;
}
@@ -281,8 +281,9 @@ const char *write_rev_file_order(const char *rev_name,
if (rev_name && adjust_shared_perm(rev_name) < 0)
die(_("failed to make %s readable"), rev_name);
- finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE |
- ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC));
+ finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA,
+ CSUM_HASH_IN_STREAM | CSUM_CLOSE |
+ ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC));
return rev_name;
}
@@ -390,7 +391,7 @@ void fixup_pack_header_footer(int pack_fd,
the_hash_algo->final_fn(partial_pack_hash, &old_hash_ctx);
the_hash_algo->final_fn(new_pack_hash, &new_hash_ctx);
write_or_die(pack_fd, new_pack_hash, the_hash_algo->rawsz);
- fsync_or_die(pack_fd, pack_name);
+ fsync_component_or_die(FSYNC_COMPONENT_PACK, pack_fd, pack_name);
}
char *index_pack_lockfile(int ip_out, int *is_well_formed)
diff --git a/read-cache.c b/read-cache.c
index f3986596623..f3539681f49 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -3060,7 +3060,7 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile,
return -1;
}
- finalize_hashfile(f, istate->oid.hash, CSUM_HASH_IN_STREAM);
+ finalize_hashfile(f, istate->oid.hash, FSYNC_COMPONENT_NONE, CSUM_HASH_IN_STREAM);
if (close_tempfile_gently(tempfile)) {
error(_("could not close '%s'"), get_tempfile_path(tempfile));
return -1;
--
gitgitgadget
^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v3 3/4] core.fsync: new option to harden the index
2021-12-09 0:57 ` [PATCH v3 0/4] " Neeraj K. Singh via GitGitGadget
2021-12-09 0:57 ` [PATCH v3 1/4] core.fsyncmethod: add writeout-only mode Neeraj Singh via GitGitGadget
2021-12-09 0:57 ` [PATCH v3 2/4] core.fsync: introduce granular fsync control Neeraj Singh via GitGitGadget
@ 2021-12-09 0:57 ` Neeraj Singh via GitGitGadget
2021-12-09 0:57 ` [PATCH v3 4/4] core.fsync: add a `derived-metadata` aggregate option Neeraj Singh via GitGitGadget
` (2 subsequent siblings)
5 siblings, 0 replies; 122+ messages in thread
From: Neeraj Singh via GitGitGadget @ 2021-12-09 0:57 UTC (permalink / raw)
To: git
Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps,
Neeraj K. Singh, Neeraj Singh
From: Neeraj Singh <neerajsi@microsoft.com>
This commit introduces the new ability for the user to harden
the index. In the event of a system crash, the index must be
durable for the user to actually find a file that has been added
to the repo and then deleted from the working tree.
We use the presence of the COMMIT_LOCK flag and absence of the
alternate_index_output as a proxy for determining whether we're
updating the persistent index of the repo or some temporary
index. We don't sync these temporary indexes.
Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
---
Documentation/config/core.txt | 1 +
cache.h | 4 +++-
config.c | 1 +
read-cache.c | 19 +++++++++++++------
4 files changed, 18 insertions(+), 7 deletions(-)
diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
index 4f1747ec871..8e5b7a795ab 100644
--- a/Documentation/config/core.txt
+++ b/Documentation/config/core.txt
@@ -561,6 +561,7 @@ core.fsync::
* `pack` hardens objects added to the repo in packfile form.
* `pack-metadata` hardens packfile bitmaps and indexes.
* `commit-graph` hardens the commit graph file.
+* `index` hardens the index when it is modified.
* `objects` is an aggregate option that includes `loose-objects`, `pack`,
`pack-metadata`, and `commit-graph`.
* `default` is an aggregate option that is equivalent to `objects,-loose-object`
diff --git a/cache.h b/cache.h
index d83fbaf2619..4dc26d7b2c9 100644
--- a/cache.h
+++ b/cache.h
@@ -996,6 +996,7 @@ enum fsync_component {
FSYNC_COMPONENT_PACK = 1 << 1,
FSYNC_COMPONENT_PACK_METADATA = 1 << 2,
FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3,
+ FSYNC_COMPONENT_INDEX = 1 << 4,
};
#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \
@@ -1010,7 +1011,8 @@ enum fsync_component {
#define FSYNC_COMPONENTS_ALL (FSYNC_COMPONENT_LOOSE_OBJECT | \
FSYNC_COMPONENT_PACK | \
FSYNC_COMPONENT_PACK_METADATA | \
- FSYNC_COMPONENT_COMMIT_GRAPH)
+ FSYNC_COMPONENT_COMMIT_GRAPH | \
+ FSYNC_COMPONENT_INDEX)
/*
diff --git a/config.c b/config.c
index 5ab381388f9..b3e7006c68e 100644
--- a/config.c
+++ b/config.c
@@ -1221,6 +1221,7 @@ static const struct fsync_component_entry {
{ "pack", FSYNC_COMPONENT_PACK },
{ "pack-metadata", FSYNC_COMPONENT_PACK_METADATA },
{ "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH },
+ { "index", FSYNC_COMPONENT_INDEX },
{ "objects", FSYNC_COMPONENTS_OBJECTS },
{ "default", FSYNC_COMPONENTS_DEFAULT },
{ "all", FSYNC_COMPONENTS_ALL },
diff --git a/read-cache.c b/read-cache.c
index f3539681f49..783cb3ea5db 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -2816,7 +2816,7 @@ static int record_ieot(void)
* rely on it.
*/
static int do_write_index(struct index_state *istate, struct tempfile *tempfile,
- int strip_extensions)
+ int strip_extensions, unsigned flags)
{
uint64_t start = getnanotime();
struct hashfile *f;
@@ -2830,6 +2830,7 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile,
struct strbuf previous_name_buf = STRBUF_INIT, *previous_name;
int drop_cache_tree = istate->drop_cache_tree;
off_t offset;
+ int csum_fsync_flag;
int ieot_entries = 1;
struct index_entry_offset_table *ieot = NULL;
int nr, nr_threads;
@@ -3060,7 +3061,13 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile,
return -1;
}
- finalize_hashfile(f, istate->oid.hash, FSYNC_COMPONENT_NONE, CSUM_HASH_IN_STREAM);
+ csum_fsync_flag = 0;
+ if (!alternate_index_output && (flags & COMMIT_LOCK))
+ csum_fsync_flag = CSUM_FSYNC;
+
+ finalize_hashfile(f, istate->oid.hash, FSYNC_COMPONENT_INDEX,
+ CSUM_HASH_IN_STREAM | csum_fsync_flag);
+
if (close_tempfile_gently(tempfile)) {
error(_("could not close '%s'"), get_tempfile_path(tempfile));
return -1;
@@ -3115,7 +3122,7 @@ static int do_write_locked_index(struct index_state *istate, struct lock_file *l
*/
trace2_region_enter_printf("index", "do_write_index", the_repository,
"%s", get_lock_file_path(lock));
- ret = do_write_index(istate, lock->tempfile, 0);
+ ret = do_write_index(istate, lock->tempfile, 0, flags);
trace2_region_leave_printf("index", "do_write_index", the_repository,
"%s", get_lock_file_path(lock));
@@ -3209,7 +3216,7 @@ static int clean_shared_index_files(const char *current_hex)
}
static int write_shared_index(struct index_state *istate,
- struct tempfile **temp)
+ struct tempfile **temp, unsigned flags)
{
struct split_index *si = istate->split_index;
int ret, was_full = !istate->sparse_index;
@@ -3219,7 +3226,7 @@ static int write_shared_index(struct index_state *istate,
trace2_region_enter_printf("index", "shared/do_write_index",
the_repository, "%s", get_tempfile_path(*temp));
- ret = do_write_index(si->base, *temp, 1);
+ ret = do_write_index(si->base, *temp, 1, flags);
trace2_region_leave_printf("index", "shared/do_write_index",
the_repository, "%s", get_tempfile_path(*temp));
@@ -3328,7 +3335,7 @@ int write_locked_index(struct index_state *istate, struct lock_file *lock,
ret = do_write_locked_index(istate, lock, flags);
goto out;
}
- ret = write_shared_index(istate, &temp);
+ ret = write_shared_index(istate, &temp, flags);
saved_errno = errno;
if (is_tempfile_active(temp))
--
gitgitgadget
^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v3 4/4] core.fsync: add a `derived-metadata` aggregate option
2021-12-09 0:57 ` [PATCH v3 0/4] " Neeraj K. Singh via GitGitGadget
` (2 preceding siblings ...)
2021-12-09 0:57 ` [PATCH v3 3/4] core.fsync: new option to harden the index Neeraj Singh via GitGitGadget
@ 2021-12-09 0:57 ` Neeraj Singh via GitGitGadget
2022-01-08 1:13 ` [PATCH v3 0/4] A design for future-proofing fsync() configuration Neeraj Singh
2022-02-01 3:33 ` [PATCH v4 " Neeraj K. Singh via GitGitGadget
5 siblings, 0 replies; 122+ messages in thread
From: Neeraj Singh via GitGitGadget @ 2021-12-09 0:57 UTC (permalink / raw)
To: git
Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps,
Neeraj K. Singh, Neeraj Singh
From: Neeraj Singh <neerajsi@microsoft.com>
This commit adds an aggregate option that currently includes the
commit-graph file and pack metadata (indexes and bitmaps).
The user may want to exclude this set from durability since they can be
recomputed from other data if they wind up corrupt or missing.
This is split out from the other patches in the series since it is
an optional nice-to-have that might be controversial.
Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
---
Documentation/config/core.txt | 6 +++---
cache.h | 7 ++++---
config.c | 1 +
3 files changed, 8 insertions(+), 6 deletions(-)
diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
index 8e5b7a795ab..21092f3a4d1 100644
--- a/Documentation/config/core.txt
+++ b/Documentation/config/core.txt
@@ -562,9 +562,9 @@ core.fsync::
* `pack-metadata` hardens packfile bitmaps and indexes.
* `commit-graph` hardens the commit graph file.
* `index` hardens the index when it is modified.
-* `objects` is an aggregate option that includes `loose-objects`, `pack`,
- `pack-metadata`, and `commit-graph`.
-* `default` is an aggregate option that is equivalent to `objects,-loose-object`
+* `objects` is an aggregate option that includes `loose-objects` and `pack`.
+* `derived-metadata` is an aggregate option that includes `pack-metadata` and `commit-graph`.
+* `default` is an aggregate option that is equivalent to `objects,derived-metadata,-loose-object`
* `all` is an aggregate option that syncs all individual components above.
core.fsyncMethod::
diff --git a/cache.h b/cache.h
index 4dc26d7b2c9..cc1c084242e 100644
--- a/cache.h
+++ b/cache.h
@@ -1004,9 +1004,10 @@ enum fsync_component {
FSYNC_COMPONENT_COMMIT_GRAPH)
#define FSYNC_COMPONENTS_OBJECTS (FSYNC_COMPONENT_LOOSE_OBJECT | \
- FSYNC_COMPONENT_PACK | \
- FSYNC_COMPONENT_PACK_METADATA | \
- FSYNC_COMPONENT_COMMIT_GRAPH)
+ FSYNC_COMPONENT_PACK)
+
+#define FSYNC_COMPONENTS_DERIVED_METADATA (FSYNC_COMPONENT_PACK_METADATA | \
+ FSYNC_COMPONENT_COMMIT_GRAPH)
#define FSYNC_COMPONENTS_ALL (FSYNC_COMPONENT_LOOSE_OBJECT | \
FSYNC_COMPONENT_PACK | \
diff --git a/config.c b/config.c
index b3e7006c68e..d9ef3ef0060 100644
--- a/config.c
+++ b/config.c
@@ -1223,6 +1223,7 @@ static const struct fsync_component_entry {
{ "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH },
{ "index", FSYNC_COMPONENT_INDEX },
{ "objects", FSYNC_COMPONENTS_OBJECTS },
+ { "derived-metadata", FSYNC_COMPONENTS_DERIVED_METADATA },
{ "default", FSYNC_COMPONENTS_DEFAULT },
{ "all", FSYNC_COMPONENTS_ALL },
};
--
gitgitgadget
^ permalink raw reply related [flat|nested] 122+ messages in thread
* Re: [PATCH v2 2/3] core.fsync: introduce granular fsync control
2021-12-09 0:14 ` Neeraj Singh
2021-12-09 0:44 ` Junio C Hamano
@ 2021-12-09 4:08 ` Ævar Arnfjörð Bjarmason
2021-12-09 6:18 ` Neeraj Singh
1 sibling, 1 reply; 122+ messages in thread
From: Ævar Arnfjörð Bjarmason @ 2021-12-09 4:08 UTC (permalink / raw)
To: Neeraj Singh
Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker,
Bagas Sanjaya, Elijah Newren, Patrick Steinhardt, Neeraj Singh
On Wed, Dec 08 2021, Neeraj Singh wrote:
> On Wed, Dec 8, 2021 at 2:17 AM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
>>
>>
>> On Tue, Dec 07 2021, Neeraj Singh wrote:
>>
>> > On Tue, Dec 7, 2021 at 5:01 AM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
>> >>
>> >>
>> >> On Tue, Dec 07 2021, Neeraj Singh via GitGitGadget wrote:
>> >>
>> >> > From: Neeraj Singh <neerajsi@microsoft.com>
>> >> >
>> >> > This commit introduces the `core.fsync` configuration
>> >> > knob which can be used to control how components of the
>> >> > repository are made durable on disk.
>> >> >
>> >> > This setting allows future extensibility of the list of
>> >> > syncable components:
>> >> > * We issue a warning rather than an error for unrecognized
>> >> > components, so new configs can be used with old Git versions.
>> >>
>> >> Looks good!
>> >>
>> >> > * We support negation, so users can choose one of the default
>> >> > aggregate options and then remove components that they don't
>> >> > want. The user would then harden any new components added in
>> >> > a Git version update.
>> >>
>> >> I think this config schema makes sense, but just a (I think important)
>> >> comment on the "how" not "what" of it. It's really much better to define
>> >> config as:
>> >>
>> >> [some]
>> >> key = value
>> >> key = value2
>> >>
>> >> Than:
>> >>
>> >> [some]
>> >> key = value,value2
>> >>
>> >> The reason is that "git config" has good support for working with
>> >> multi-valued stuff, so you can do e.g.:
>> >>
>> >> git config --get-all -z some.key
>> >>
>> >> And you can easily (re)set such config e.g. with --replace-all etc., but
>> >> for comma-delimited you (and users) need to do all that work themselves.
>> >>
>> >> Similarly instead of:
>> >>
>> >> some.key = want-this
>> >> some.key = -not-this
>> >> some.key = but-want-this
>> >>
>> >> I think it's better to just have two lists, one inclusive another
>> >> exclusive. E.g. see "log.decorate" and "log.excludeDecoration",
>> >> "transfer.hideRefs"
>> >>
>> >> Which would mean:
>> >>
>> >> core.fsync = want-this
>> >> core.fsyncExcludes = -not-this
>> >>
>> >> For some value of "fsyncExcludes", maybe "noFsync"? Anyway, just a
>> >> suggestion on making this easier for users & the implementation.
>> >>
>> >
>> > Maybe there's some way to handle this I'm unaware of, but a
>> > disadvantage of your multi-valued config proposal is that it's harder,
>> > for example, for a per-repo config store to reasonably override a
>> > per-user config store. With the configuration scheme as-is, I can
>> > have a per-user setting like `core.fsync=all` which covers my typical
>> > repos, but then have a maintainer repo with a private setting of
>> > `core.fsync=none` to speed up cases where I'm mostly working with
>> > other people's changes that are backed up in email or server-side
>> > repos. The latter setting conveniently overrides the former setting
>> > in all aspects.
>>
>> Even if you turn just your comma-delimited proposal into a list proposal
>> can't we just say that the last one wins? Then it won't matter what cmae
>> before, you'd specify "core.fsync=none" in your local .git/config.
>>
>> But this is also a general issue with a bunch of things in git's config
>> space. I'd rather see us use the list-based values and just come up with
>> some general way to reset them that works for all keys, rather than
>> regretting having comma-delimited values that'll be harder to work with
>> & parse, which will be a legacy wart if/when we come up with a way to
>> say "reset all previous settings".
>>
>> > Also, with the core.fsync and core.fsyncExcludes, how would you spell
>> > "don't sync anything"? Would you still have the aggregate options.?
>>
>> core.fsyncExcludes = *
>>
>> I.e. the same as the core.fsync=none above, anyway I still like the
>> wildcard thing below a bit more...
>
> I'm not going to take this feedback unless there are additional votes
> from the Git community in this direction. I make the claim that
> single-valued comma-separated config lists are easier to work with in
> the existing Git infrastructure.
Easier in what sense? I showed examples of how "git-config" trivially
works with multi-valued config, but for comma-delimited you'll need to
write your own shellscript boilerplate around simple things like adding
values, removing existing ones etc.
I.e. you can use --add, --unset, --unset-all, --get, --get-all etc.
> parsing code for the core.whitespace variable and users are used to
> this syntax there. There are several other comma-separated lists in
> the config space, so this construct has precedence and will be with
> Git for some time.
That's not really an argument either way for why we'd pick X over Y for
something new. We've got some comma-delimited, some multi-value (I'm
fairly sure we have more multi-value).
> Also, fsync configurations aren't composable like
> some other configurations may be. It makes sense to have a holistic
> singular fsync configuration, which is best represented by a single
> variable.
What's a "variable" here? We call these "keys", you can have a
single-value key like user.name that you get with --get, or a
multi-value key like say branch.<name>.merge or push.pushOption that
you'd get with --get-all.
I think you may be referring to either not wanting these to be
"inherited" (which is not really a think we do for anything else in
config), or lacking the ability to "reset".
For the latter if that's absolutely needed we could just use the same
trick as "diff.wsErrorHighlight" uses of making an empty value "reset"
the list, and you'd get better "git config" support for editing it.
>> >> > This also supports the common request of doing absolutely no
>> >> > fysncing with the `core.fsync=none` value, which is expected
>> >> > to make the test suite faster.
>> >>
>> >> Let's just use the git_parse_maybe_bool() or git_parse_maybe_bool_text()
>> >> so we'll accept "false", "off", "no" like most other such config?
>> >
>> > Junio's previous feedback when discussing batch mode [1] was to offer
>> > less flexibility when parsing new values of these configuration
>> > options. I agree with his statement that "making machine-readable
>> > tokens be spelled in different ways is a 'disease'." I'd like to
>> > leave this as-is so that the documentation can clearly state the exact
>> > set of allowable values.
>> >
>> > [1] https://lore.kernel.org/git/xmqqr1dqzyl7.fsf@gitster.g/
>>
>> I think he's talking about batch, Batch, BATCH, bAtCh etc. there. But
>> the "maybe bool" is a stanard pattern we use.
>>
>> I don't think we'd call one of these 0, off, no or false etc. to avoid
>> confusion, so then you can use git_parse_maybe_...()
>
> I don't see the advantage of having multiple ways of specifying
> "none". The user can read the doc and know exactly what to write. If
> they write something unallowable, they get a clear warning and they
> can read the doc again to figure out what to write. This isn't a
> boolean options at all, so why should we entertain bool-like ways of
> spelling it?
It's not boolean, it's multi-value and one of the values includes a true
or false boolean value. You just spell it "none".
I think both this and your comment above suggest that you think there's
no point in this because you haven't interacted with/used "git config"
as a command line or API mechanism, but have just hand-crafted config
files.
That's fair enough, but there's a lot of tooling that benefits from the
latter.
E.g.:
$ git -c core.fsync=off config --type=bool core.fsync
false
$ git -c core.fsync=blah config --type=bool core.fsync
fatal: bad boolean config value 'blah' for 'core.fsync'
Here we can get 'git config' to normalize what you call 'none', and you
can tell via exit codes/normalization if it's "false". But if you invent
a new term for "false" you can't do that as easily.
We have various historical keys that take odd exceptions to that,
e.g. "never", but unless we have a good reason to let's not invent more
exceptions.
>> >> > Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
>> >> > ---
>> >> > Documentation/config/core.txt | 27 +++++++++----
>> >> > builtin/fast-import.c | 2 +-
>> >> > builtin/index-pack.c | 4 +-
>> >> > builtin/pack-objects.c | 8 ++--
>> >> > bulk-checkin.c | 5 ++-
>> >> > cache.h | 39 +++++++++++++++++-
>> >> > commit-graph.c | 3 +-
>> >> > config.c | 76 ++++++++++++++++++++++++++++++++++-
>> >> > csum-file.c | 5 ++-
>> >> > csum-file.h | 3 +-
>> >> > environment.c | 1 +
>> >> > midx.c | 3 +-
>> >> > object-file.c | 3 +-
>> >> > pack-bitmap-write.c | 3 +-
>> >> > pack-write.c | 13 +++---
>> >> > read-cache.c | 2 +-
>> >> > 16 files changed, 164 insertions(+), 33 deletions(-)
>> >> >
>> >> > diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
>> >> > index dbb134f7136..4f1747ec871 100644
>> >> > --- a/Documentation/config/core.txt
>> >> > +++ b/Documentation/config/core.txt
>> >> > @@ -547,6 +547,25 @@ core.whitespace::
>> >> > is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent`
>> >> > errors. The default tab width is 8. Allowed values are 1 to 63.
>> >> >
>> >> > +core.fsync::
>> >> > + A comma-separated list of parts of the repository which should be
>> >> > + hardened via the core.fsyncMethod when created or modified. You can
>> >> > + disable hardening of any component by prefixing it with a '-'. Later
>> >> > + items take precedence over earlier ones in the list. For example,
>> >> > + `core.fsync=all,-pack-metadata` means "harden everything except pack
>> >> > + metadata." Items that are not hardened may be lost in the event of an
>> >> > + unclean system shutdown.
>> >> > ++
>> >> > +* `none` disables fsync completely. This must be specified alone.
>> >> > +* `loose-object` hardens objects added to the repo in loose-object form.
>> >> > +* `pack` hardens objects added to the repo in packfile form.
>> >> > +* `pack-metadata` hardens packfile bitmaps and indexes.
>> >> > +* `commit-graph` hardens the commit graph file.
>> >> > +* `objects` is an aggregate option that includes `loose-objects`, `pack`,
>> >> > + `pack-metadata`, and `commit-graph`.
>> >> > +* `default` is an aggregate option that is equivalent to `objects,-loose-object`
>> >> > +* `all` is an aggregate option that syncs all individual components above.
>> >> > +
>> >>
>> >> It's probably a *bit* more work to set up, but I wonder if this wouldn't
>> >> be simpler if we just said (and this is partially going against what I
>> >> noted above):
>> >>
>> >> == BEGIN DOC
>> >>
>> >> core.fsync is a multi-value config variable where each item is a
>> >> pathspec that'll get matched the same way as 'git-ls-files' et al.
>> >>
>> >> When we sync pretend that a path like .git/objects/de/adbeef... is
>> >> relative to the top-level of the git
>> >> directory. E.g. "objects/de/adbeaf.." or "objects/pack/...".
>> >>
>> >> You can then supply a list of wildcards and exclusions to configure
>> >> syncing. or "false", "off" etc. to turn it off. These are synonymous
>> >> with:
>> >>
>> >> ; same as "false"
>> >> core.fsync = ":!*"
>> >>
>> >> Or:
>> >>
>> >> ; same as "true"
>> >> core.fsync = "*"
>> >>
>> >> Or, to selectively sync some things and not others:
>> >>
>> >> ;; Sync objects, but not "info"
>> >> core.fsync = ":!objects/info/**"
>> >> core.fsync = "objects/**"
>> >>
>> >> See gitrepository-layout(5) for details about what sort of paths you
>> >> might be expected to match. Not all paths listed there will go through
>> >> this mechanism (e.g. currently objects do, but nothing to do with config
>> >> does).
>> >>
>> >> We can and will match this against "fake paths", e.g. when writing out
>> >> packs we may match against just the string "objects/pack", we're not
>> >> going to re-check if every packfile we're writing matches your globs,
>> >> ditto for loose objects. Be reasonable!
>> >>
>> >> This metharism is intended as a shorthand that provides some flexibility
>> >> when fsyncing, while not forcing git to come up with labels for all
>> >> paths the git dir, or to support crazyness like "objects/de/adbeef*"
>> >>
>> >> More paths may be added or removed in the future, and we make no
>> >> promises that we won't move things around, so if in doubt use
>> >> e.g. "true" or a wide pattern match like "objects/**". When in doubt
>> >> stick to the golden path of examples provided in this documentation.
>> >>
>> >> == END DOC
>> >>
>> >>
>> >> It's a tad more complex to set up, but I wonder if that isn't worth
>> >> it. It nicely gets around any current and future issues of deciding what
>> >> labels such as "loose-object" etc. to pick, as well as slotting into an
>> >> existing method of doing exclude/include lists.
>> >>
>> >
>> > I think this proposal is a lot of complexity to avoid coming up with a
>> > new name for syncable things as they are added to Git. A path based
>> > mechanism makes it hard to document for the (advanced) user what the
>> > full set of things is and how it might change from release to release.
>> > I think the current core.fsync scheme is a bit easier to understand,
>> > query, and extend.
>>
>> We document it in gitrepository-layout(5). Yeah it has some
>> disadvantages, but one advantage is that you could make the
>> composability easy. I.e. if last exclude wins then a setting of:
>>
>> core.fsync = ":!*"
>> core.fsync = "objects/**"
>>
>> Would reset all previous matches & only match objects/**.
>>
>
> The value of changing this is predicated on taking your previous
> multi-valued config proposal, which I'm still not at all convinced
> about.
They're orthagonal. I.e. you get benefits from multi-value with or
without this globbing mechanism.
In any case, I don't feel strongly about/am really advocating this
globbing mechanism. I just wondered if it wouldn't make things simpler
since it would sidestep the need to create any sort of categories for
subsets of gitrepository-layout(5), but maybe not...
> The schema in the current (v1-v2) version of the patch already
> includes an example of extending the list of syncable things, and
> Patrick Steinhardt made it clear that he feels comfortable adding
> 'refs' to the same schema in a future change.
>
> I'll also emphasize that we're talking about a non-functional,
> relatively corner-case behavioral configuration. These values don't
> change how git's interface behaves except when the system crashes
> during a git command or shortly after one completes.
That may be something some OS's promise, but it's not something fsync()
or POSIX promises. I.e. you might write a ref, but unless you fsync and
the relevant dir entries another process might not see the update, crash
or not.
That's an aside from these config design questions, and I think
most/(all?) OS's anyone cares about these days tend to make that
implicit promise as part of their VFS behavior, but we should probably
design such an interface to fsync() with such pedantic portability in
mind.
> While you may not personally love the proposed configuration
> interface, I'd want your view on some questions:
> 1. Is it easy for the (advanced) user to set a configuration?
> 2. Is it easy for the (advanced) user to see what was configured?
> 3. Is it easy for the Git community to build on this as we want to add
> things to the list of things to sync?
> a) Is there a good best practice configuration so that people can
> avoid losing integrity for new stuff that they are intending to sync.
> b) If someone has a custom configuration, can that custom
> configuration do something reasonable as they upgrade versions of Git?
> ** In response to this question, I might see some value
> in adding a 'derived-metadata' aggregate that can be disabled so that
> a custom configuration can exclude those as they change version to
> version.
> c) Is it too much maintenance overhead to consider how to present
> this configuration knob for any new hashfile or other datafile in the
> git repo?
> 4. Is there a good path forward to change the default syncable set,
> both in git-for-windows and in Git for other platforms?
I'm not really sure this globbing this is a good idea, as noted above
just a suggestion etc.
As noted there it just gets you out of the business of re-defining
gitrepository-layout(5), and assuming too much in advance about certain
use-cases.
E.g. even "refs" might be too broad for some. I don't tend to be I/O
limited, but I could see how someone who would be would care about
refs/heads but not refs/remotes, or want to exclude logs/* but not the
refs updates themselves etc.
>> >> > diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
>> >> > index 857be7826f3..916c55d6ce9 100644
>> >> > --- a/builtin/pack-objects.c
>> >> > +++ b/builtin/pack-objects.c
>> >> > @@ -1204,11 +1204,13 @@ static void write_pack_file(void)
>> >> > * If so, rewrite it like in fast-import
>> >> > */
>> >> > if (pack_to_stdout) {
>> >> > - finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_CLOSE);
>> >> > + finalize_hashfile(f, hash, FSYNC_COMPONENT_NONE,
>> >> > + CSUM_HASH_IN_STREAM | CSUM_CLOSE);
>> >>
>> >> Not really related to this per-se, but since you're touching the API
>> >> everything goes through I wonder if callers should just always try to
>> >> fsync, and we can just catch EROFS and EINVAL in the wrapper if someone
>> >> tries to flush stdout, or catch the fd at that lower level.
>> >>
>> >> Or maybe there's a good reason for this...
>> >
>> > It's platform dependent, but I'd expect fsync would do something for
>> > pipes or stdout redirected to a file. In these cases we really don't
>> > want to fsync since we have no idea what we're talking to and we're
>> > potentially worsening performance for probably no benefit.
>>
>> Yeah maybe we should just leave it be.
>>
>> I'd think the C library returning EINVAL would be a trivial performance
>> cost though.
>>
>> It just seemed odd to hardcode assumptions about what can and can't be
>> synced when the POSIX defined function will also tell us that.
>>
>
> Redirecting stdout to a file seems like a common usage for this
> command. That would definitely be fsyncable, but Git has no idea what
> its proper category is since there's no way to know the purpose or
> lifetime of the packfile. I'm going to leave this be, because I'd
> posit that "can it be fsynced?" is not the same as "should it be
> fsynced?". The latter question can't be answered for stdout.
As noted this was just an aside, and I don't even know if any OS would
do anything meaningful with an fsync() of such a FD anyway.
I just don't see why we wouldn't say:
1. We're syncing this category of thing
2. Try it
3. If fsync returns "can't fsync that sort of thing" move on
As opposed to trying to shortcut #3 by doing the detection ourselves.
I.e. maybe there was a good reason, but it seemed to be some easy
potential win for more simplification, since you were re-doing and
simplifying some of the interface anyway...
>>
>> >> > [...]
>> >> > +/*
>> >> > + * These values are used to help identify parts of a repository to fsync.
>> >> > + * FSYNC_COMPONENT_NONE identifies data that will not be a persistent part of the
>> >> > + * repository and so shouldn't be fsynced.
>> >> > + */
>> >> > +enum fsync_component {
>> >> > + FSYNC_COMPONENT_NONE = 0,
>> >>
>> >> I haven't read ahead much but in most other such cases we don't define
>> >> the "= 0", just start at 1<<0, then check the flags elsewhere...
>> >>
>> >> > +static const struct fsync_component_entry {
>> >> > + const char *name;
>> >> > + enum fsync_component component_bits;
>> >> > +} fsync_component_table[] = {
>> >> > + { "loose-object", FSYNC_COMPONENT_LOOSE_OBJECT },
>> >> > + { "pack", FSYNC_COMPONENT_PACK },
>> >> > + { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA },
>> >> > + { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH },
>> >> > + { "objects", FSYNC_COMPONENTS_OBJECTS },
>> >> > + { "default", FSYNC_COMPONENTS_DEFAULT },
>> >> > + { "all", FSYNC_COMPONENTS_ALL },
>> >> > +};
>> >> > +
>> >> > +static enum fsync_component parse_fsync_components(const char *var, const char *string)
>> >> > +{
>> >> > + enum fsync_component output = 0;
>> >> > +
>> >> > + if (!strcmp(string, "none"))
>> >> > + return output;
>> >> > +
>> >> > + while (string) {
>> >> > + int i;
>> >> > + size_t len;
>> >> > + const char *ep;
>> >> > + int negated = 0;
>> >> > + int found = 0;
>> >> > +
>> >> > + string = string + strspn(string, ", \t\n\r");
>> >>
>> >> Aside from the "use a list" isn't this hardcoding some windows-specific
>> >> assumptions with \n\r? Maybe not...
>> >
>> > I shamelessly stole this code from parse_whitespace_rule. I thought
>> > about making a helper to be called by both functions, but the amount
>> > of state going into and out of the wrapper via arguments was
>> > substantial and seemed to negate the benefit of deduplication.
>>
>> FWIW string_list_split() is easier to work with in those cases, or at
>> least I think so...
>
> This code runs at startup for a variable that may be present on some
> installations. The nice property of the current patch's code is that
> it's already a well-tested pattern that doesn't do any allocations as
> it's working, unlike string_list_split().
Multi-value config would also get you fewer allocations :)
Anyway, I mainly meant to point out that for stuff like this it's fine
to optimize it for ease rather than micro-optimize allocations. Those
really aren't a bottleneck on this scale.
Even in that case there's string_list_split_in_place(), which can be a
bit nicer than manual C-string fiddling.
> I hope you know that I appreciate your review feedback, even though
> I'm pushing back on most of it so far this round. I'll be sending v3
> to the list soon after giving it another look over.
Sure, no worries. Just hoping to help. If you go for something different
etc. that's fine. Just hoping to bridge the gap in some knowledge /
offer potentially interesting suggestions (some of which may be dead
ends, like the config glob thing...).
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v2 2/3] core.fsync: introduce granular fsync control
2021-12-09 4:08 ` Ævar Arnfjörð Bjarmason
@ 2021-12-09 6:18 ` Neeraj Singh
2022-01-18 23:50 ` Neeraj Singh
2022-01-19 14:52 ` Ævar Arnfjörð Bjarmason
0 siblings, 2 replies; 122+ messages in thread
From: Neeraj Singh @ 2021-12-09 6:18 UTC (permalink / raw)
To: Ævar Arnfjörð Bjarmason
Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker,
Bagas Sanjaya, Elijah Newren, Patrick Steinhardt, Neeraj Singh
On Wed, Dec 8, 2021 at 8:46 PM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
>
>
> On Wed, Dec 08 2021, Neeraj Singh wrote:
>
> > On Wed, Dec 8, 2021 at 2:17 AM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
> >>
> >>
> >> On Tue, Dec 07 2021, Neeraj Singh wrote:
> >>
> >> > On Tue, Dec 7, 2021 at 5:01 AM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
> >> >>
> >> >>
> >> >> On Tue, Dec 07 2021, Neeraj Singh via GitGitGadget wrote:
> >> >>
> >> >> > From: Neeraj Singh <neerajsi@microsoft.com>
> >> >> >
> >> >> > This commit introduces the `core.fsync` configuration
> >> >> > knob which can be used to control how components of the
> >> >> > repository are made durable on disk.
> >> >> >
> >> >> > This setting allows future extensibility of the list of
> >> >> > syncable components:
> >> >> > * We issue a warning rather than an error for unrecognized
> >> >> > components, so new configs can be used with old Git versions.
> >> >>
> >> >> Looks good!
> >> >>
> >> >> > * We support negation, so users can choose one of the default
> >> >> > aggregate options and then remove components that they don't
> >> >> > want. The user would then harden any new components added in
> >> >> > a Git version update.
> >> >>
> >> >> I think this config schema makes sense, but just a (I think important)
> >> >> comment on the "how" not "what" of it. It's really much better to define
> >> >> config as:
> >> >>
> >> >> [some]
> >> >> key = value
> >> >> key = value2
> >> >>
> >> >> Than:
> >> >>
> >> >> [some]
> >> >> key = value,value2
> >> >>
> >> >> The reason is that "git config" has good support for working with
> >> >> multi-valued stuff, so you can do e.g.:
> >> >>
> >> >> git config --get-all -z some.key
> >> >>
> >> >> And you can easily (re)set such config e.g. with --replace-all etc., but
> >> >> for comma-delimited you (and users) need to do all that work themselves.
> >> >>
> >> >> Similarly instead of:
> >> >>
> >> >> some.key = want-this
> >> >> some.key = -not-this
> >> >> some.key = but-want-this
> >> >>
> >> >> I think it's better to just have two lists, one inclusive another
> >> >> exclusive. E.g. see "log.decorate" and "log.excludeDecoration",
> >> >> "transfer.hideRefs"
> >> >>
> >> >> Which would mean:
> >> >>
> >> >> core.fsync = want-this
> >> >> core.fsyncExcludes = -not-this
> >> >>
> >> >> For some value of "fsyncExcludes", maybe "noFsync"? Anyway, just a
> >> >> suggestion on making this easier for users & the implementation.
> >> >>
> >> >
> >> > Maybe there's some way to handle this I'm unaware of, but a
> >> > disadvantage of your multi-valued config proposal is that it's harder,
> >> > for example, for a per-repo config store to reasonably override a
> >> > per-user config store. With the configuration scheme as-is, I can
> >> > have a per-user setting like `core.fsync=all` which covers my typical
> >> > repos, but then have a maintainer repo with a private setting of
> >> > `core.fsync=none` to speed up cases where I'm mostly working with
> >> > other people's changes that are backed up in email or server-side
> >> > repos. The latter setting conveniently overrides the former setting
> >> > in all aspects.
> >>
> >> Even if you turn just your comma-delimited proposal into a list proposal
> >> can't we just say that the last one wins? Then it won't matter what cmae
> >> before, you'd specify "core.fsync=none" in your local .git/config.
> >>
> >> But this is also a general issue with a bunch of things in git's config
> >> space. I'd rather see us use the list-based values and just come up with
> >> some general way to reset them that works for all keys, rather than
> >> regretting having comma-delimited values that'll be harder to work with
> >> & parse, which will be a legacy wart if/when we come up with a way to
> >> say "reset all previous settings".
> >>
> >> > Also, with the core.fsync and core.fsyncExcludes, how would you spell
> >> > "don't sync anything"? Would you still have the aggregate options.?
> >>
> >> core.fsyncExcludes = *
> >>
> >> I.e. the same as the core.fsync=none above, anyway I still like the
> >> wildcard thing below a bit more...
> >
> > I'm not going to take this feedback unless there are additional votes
> > from the Git community in this direction. I make the claim that
> > single-valued comma-separated config lists are easier to work with in
> > the existing Git infrastructure.
>
> Easier in what sense? I showed examples of how "git-config" trivially
> works with multi-valued config, but for comma-delimited you'll need to
> write your own shellscript boilerplate around simple things like adding
> values, removing existing ones etc.
>
> I.e. you can use --add, --unset, --unset-all, --get, --get-all etc.
>
I see what you're saying for cases where someone would want to set a
core.fsync setting that's derived from the user's current config. But
I'm guessing that the dominant use case is someone setting a new fsync
configuration that achieves some atomic goal with respect to a given
repository. Like "this is a throwaway, so sync nothing" or "this is
really important, so sync all objects and refs and the index".
> > parsing code for the core.whitespace variable and users are used to
> > this syntax there. There are several other comma-separated lists in
> > the config space, so this construct has precedence and will be with
> > Git for some time.
>
> That's not really an argument either way for why we'd pick X over Y for
> something new. We've got some comma-delimited, some multi-value (I'm
> fairly sure we have more multi-value).
>
My main point here is that there's precedent for patch's current exact
schema for a config in the same core config leaf of the Documentation.
It seems from your comments that we'd have to invent and document some
new convention for "reset" of a multi-valued config. So you're
suggesting that I solve an extra set of problems to get this change
in. Just want to remind you that my personal itch to scratch is to
get the underlying mechanism in so that git-for-windows can set its
default setting to batch mode. I'm not expecting many users to
actually configure this setting to any non-default value.
> > Also, fsync configurations aren't composable like
> > some other configurations may be. It makes sense to have a holistic
> > singular fsync configuration, which is best represented by a single
> > variable.
>
> What's a "variable" here? We call these "keys", you can have a
> single-value key like user.name that you get with --get, or a
> multi-value key like say branch.<name>.merge or push.pushOption that
> you'd get with --get-all.
Yeah, I meant "key". I conflated the config key with the underlying
global variable in git.
> I think you may be referring to either not wanting these to be
> "inherited" (which is not really a think we do for anything else in
> config), or lacking the ability to "reset".
>
> For the latter if that's absolutely needed we could just use the same
> trick as "diff.wsErrorHighlight" uses of making an empty value "reset"
> the list, and you'd get better "git config" support for editing it.
>
My reading of the code is that diff.wsErrorHighlight is a comma
separated list and not a multi-valued config. Actually I haven't yet
found an existing multi-valued config (not sure how to grep for it).
> >> >> > This also supports the common request of doing absolutely no
> >> >> > fysncing with the `core.fsync=none` value, which is expected
> >> >> > to make the test suite faster.
> >> >>
> >> >> Let's just use the git_parse_maybe_bool() or git_parse_maybe_bool_text()
> >> >> so we'll accept "false", "off", "no" like most other such config?
> >> >
> >> > Junio's previous feedback when discussing batch mode [1] was to offer
> >> > less flexibility when parsing new values of these configuration
> >> > options. I agree with his statement that "making machine-readable
> >> > tokens be spelled in different ways is a 'disease'." I'd like to
> >> > leave this as-is so that the documentation can clearly state the exact
> >> > set of allowable values.
> >> >
> >> > [1] https://lore.kernel.org/git/xmqqr1dqzyl7.fsf@gitster.g/
> >>
> >> I think he's talking about batch, Batch, BATCH, bAtCh etc. there. But
> >> the "maybe bool" is a stanard pattern we use.
> >>
> >> I don't think we'd call one of these 0, off, no or false etc. to avoid
> >> confusion, so then you can use git_parse_maybe_...()
> >
> > I don't see the advantage of having multiple ways of specifying
> > "none". The user can read the doc and know exactly what to write. If
> > they write something unallowable, they get a clear warning and they
> > can read the doc again to figure out what to write. This isn't a
> > boolean options at all, so why should we entertain bool-like ways of
> > spelling it?
>
> It's not boolean, it's multi-value and one of the values includes a true
> or false boolean value. You just spell it "none".
>
> I think both this and your comment above suggest that you think there's
> no point in this because you haven't interacted with/used "git config"
> as a command line or API mechanism, but have just hand-crafted config
> files.
>
> That's fair enough, but there's a lot of tooling that benefits from the
> latter.
My batch mode perf tests (on github, not yet submitted to the list)
use `git -c core.fsync=<foo>` to set up a per-process config. I
haven't used the `git config` writing support in a while, so I haven't
deeply thought about that. However, I favor simplifying the use case
of "atomically" setting a new holistic core.fsync value versus the use
case of deriving a new core.fsync value from the preexisting value.
> E.g.:
>
> $ git -c core.fsync=off config --type=bool core.fsync
> false
> $ git -c core.fsync=blah config --type=bool core.fsync
> fatal: bad boolean config value 'blah' for 'core.fsync'
>
> Here we can get 'git config' to normalize what you call 'none', and you
> can tell via exit codes/normalization if it's "false". But if you invent
> a new term for "false" you can't do that as easily.
>
> We have various historical keys that take odd exceptions to that,
> e.g. "never", but unless we have a good reason to let's not invent more
> exceptions.
>
> >> >> > Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
> >> >> > ---
> >> >> > Documentation/config/core.txt | 27 +++++++++----
> >> >> > builtin/fast-import.c | 2 +-
> >> >> > builtin/index-pack.c | 4 +-
> >> >> > builtin/pack-objects.c | 8 ++--
> >> >> > bulk-checkin.c | 5 ++-
> >> >> > cache.h | 39 +++++++++++++++++-
> >> >> > commit-graph.c | 3 +-
> >> >> > config.c | 76 ++++++++++++++++++++++++++++++++++-
> >> >> > csum-file.c | 5 ++-
> >> >> > csum-file.h | 3 +-
> >> >> > environment.c | 1 +
> >> >> > midx.c | 3 +-
> >> >> > object-file.c | 3 +-
> >> >> > pack-bitmap-write.c | 3 +-
> >> >> > pack-write.c | 13 +++---
> >> >> > read-cache.c | 2 +-
> >> >> > 16 files changed, 164 insertions(+), 33 deletions(-)
> >> >> >
> >> >> > diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
> >> >> > index dbb134f7136..4f1747ec871 100644
> >> >> > --- a/Documentation/config/core.txt
> >> >> > +++ b/Documentation/config/core.txt
> >> >> > @@ -547,6 +547,25 @@ core.whitespace::
> >> >> > is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent`
> >> >> > errors. The default tab width is 8. Allowed values are 1 to 63.
> >> >> >
> >> >> > +core.fsync::
> >> >> > + A comma-separated list of parts of the repository which should be
> >> >> > + hardened via the core.fsyncMethod when created or modified. You can
> >> >> > + disable hardening of any component by prefixing it with a '-'. Later
> >> >> > + items take precedence over earlier ones in the list. For example,
> >> >> > + `core.fsync=all,-pack-metadata` means "harden everything except pack
> >> >> > + metadata." Items that are not hardened may be lost in the event of an
> >> >> > + unclean system shutdown.
> >> >> > ++
> >> >> > +* `none` disables fsync completely. This must be specified alone.
> >> >> > +* `loose-object` hardens objects added to the repo in loose-object form.
> >> >> > +* `pack` hardens objects added to the repo in packfile form.
> >> >> > +* `pack-metadata` hardens packfile bitmaps and indexes.
> >> >> > +* `commit-graph` hardens the commit graph file.
> >> >> > +* `objects` is an aggregate option that includes `loose-objects`, `pack`,
> >> >> > + `pack-metadata`, and `commit-graph`.
> >> >> > +* `default` is an aggregate option that is equivalent to `objects,-loose-object`
> >> >> > +* `all` is an aggregate option that syncs all individual components above.
> >> >> > +
> >> >>
> >> >> It's probably a *bit* more work to set up, but I wonder if this wouldn't
> >> >> be simpler if we just said (and this is partially going against what I
> >> >> noted above):
> >> >>
> >> >> == BEGIN DOC
> >> >>
> >> >> core.fsync is a multi-value config variable where each item is a
> >> >> pathspec that'll get matched the same way as 'git-ls-files' et al.
> >> >>
> >> >> When we sync pretend that a path like .git/objects/de/adbeef... is
> >> >> relative to the top-level of the git
> >> >> directory. E.g. "objects/de/adbeaf.." or "objects/pack/...".
> >> >>
> >> >> You can then supply a list of wildcards and exclusions to configure
> >> >> syncing. or "false", "off" etc. to turn it off. These are synonymous
> >> >> with:
> >> >>
> >> >> ; same as "false"
> >> >> core.fsync = ":!*"
> >> >>
> >> >> Or:
> >> >>
> >> >> ; same as "true"
> >> >> core.fsync = "*"
> >> >>
> >> >> Or, to selectively sync some things and not others:
> >> >>
> >> >> ;; Sync objects, but not "info"
> >> >> core.fsync = ":!objects/info/**"
> >> >> core.fsync = "objects/**"
> >> >>
> >> >> See gitrepository-layout(5) for details about what sort of paths you
> >> >> might be expected to match. Not all paths listed there will go through
> >> >> this mechanism (e.g. currently objects do, but nothing to do with config
> >> >> does).
> >> >>
> >> >> We can and will match this against "fake paths", e.g. when writing out
> >> >> packs we may match against just the string "objects/pack", we're not
> >> >> going to re-check if every packfile we're writing matches your globs,
> >> >> ditto for loose objects. Be reasonable!
> >> >>
> >> >> This metharism is intended as a shorthand that provides some flexibility
> >> >> when fsyncing, while not forcing git to come up with labels for all
> >> >> paths the git dir, or to support crazyness like "objects/de/adbeef*"
> >> >>
> >> >> More paths may be added or removed in the future, and we make no
> >> >> promises that we won't move things around, so if in doubt use
> >> >> e.g. "true" or a wide pattern match like "objects/**". When in doubt
> >> >> stick to the golden path of examples provided in this documentation.
> >> >>
> >> >> == END DOC
> >> >>
> >> >>
> >> >> It's a tad more complex to set up, but I wonder if that isn't worth
> >> >> it. It nicely gets around any current and future issues of deciding what
> >> >> labels such as "loose-object" etc. to pick, as well as slotting into an
> >> >> existing method of doing exclude/include lists.
> >> >>
> >> >
> >> > I think this proposal is a lot of complexity to avoid coming up with a
> >> > new name for syncable things as they are added to Git. A path based
> >> > mechanism makes it hard to document for the (advanced) user what the
> >> > full set of things is and how it might change from release to release.
> >> > I think the current core.fsync scheme is a bit easier to understand,
> >> > query, and extend.
> >>
> >> We document it in gitrepository-layout(5). Yeah it has some
> >> disadvantages, but one advantage is that you could make the
> >> composability easy. I.e. if last exclude wins then a setting of:
> >>
> >> core.fsync = ":!*"
> >> core.fsync = "objects/**"
> >>
> >> Would reset all previous matches & only match objects/**.
> >>
> >
> > The value of changing this is predicated on taking your previous
> > multi-valued config proposal, which I'm still not at all convinced
> > about.
>
> They're orthagonal. I.e. you get benefits from multi-value with or
> without this globbing mechanism.
>
> In any case, I don't feel strongly about/am really advocating this
> globbing mechanism. I just wondered if it wouldn't make things simpler
> since it would sidestep the need to create any sort of categories for
> subsets of gitrepository-layout(5), but maybe not...
>
> > The schema in the current (v1-v2) version of the patch already
> > includes an example of extending the list of syncable things, and
> > Patrick Steinhardt made it clear that he feels comfortable adding
> > 'refs' to the same schema in a future change.
> >
> > I'll also emphasize that we're talking about a non-functional,
> > relatively corner-case behavioral configuration. These values don't
> > change how git's interface behaves except when the system crashes
> > during a git command or shortly after one completes.
>
> That may be something some OS's promise, but it's not something fsync()
> or POSIX promises. I.e. you might write a ref, but unless you fsync and
> the relevant dir entries another process might not see the update, crash
> or not.
>
I haven't seen any indication that POSIX requires an fsync for
visiblity within a running system. I looked at the doc for open,
write, and fsync, and saw no indication that it's posix compliant to
require an fsync for visibility. I think an OS that required fsync
for cross-process visiblity would fail to run Git for a myriad of
other reasons and would likely lose all its users. I'm curious where
you've seen documentation that allows such unhelpful behavior?
> That's an aside from these config design questions, and I think
> most/(all?) OS's anyone cares about these days tend to make that
> implicit promise as part of their VFS behavior, but we should probably
> design such an interface to fsync() with such pedantic portability in
> mind.
Why? To be rude to such a hypothetical system, if a system were so
insanely designed, it would be nuts to support it.
> > While you may not personally love the proposed configuration
> > interface, I'd want your view on some questions:
> > 1. Is it easy for the (advanced) user to set a configuration?
> > 2. Is it easy for the (advanced) user to see what was configured?
> > 3. Is it easy for the Git community to build on this as we want to add
> > things to the list of things to sync?
> > a) Is there a good best practice configuration so that people can
> > avoid losing integrity for new stuff that they are intending to sync.
> > b) If someone has a custom configuration, can that custom
> > configuration do something reasonable as they upgrade versions of Git?
> > ** In response to this question, I might see some value
> > in adding a 'derived-metadata' aggregate that can be disabled so that
> > a custom configuration can exclude those as they change version to
> > version.
> > c) Is it too much maintenance overhead to consider how to present
> > this configuration knob for any new hashfile or other datafile in the
> > git repo?
> > 4. Is there a good path forward to change the default syncable set,
> > both in git-for-windows and in Git for other platforms?
>
> I'm not really sure this globbing this is a good idea, as noted above
> just a suggestion etc.
>
> As noted there it just gets you out of the business of re-defining
> gitrepository-layout(5), and assuming too much in advance about certain
> use-cases.
>
> E.g. even "refs" might be too broad for some. I don't tend to be I/O
> limited, but I could see how someone who would be would care about
> refs/heads but not refs/remotes, or want to exclude logs/* but not the
> refs updates themselves etc.
This use-case is interesting (distinguishing remote refs from local
refs). I think the difficulty of verifying (for even an advanced
user) that the right fsyncing is actually happening still puts me on
the side of having a carefully curated and documented set of syncable
things rather than a file-path-based mechanism.
Is this meaningful in the presumably nearby future world of the refsdb
backend? Is that somehow split by remote versus local?
> >> >> > diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
> >> >> > index 857be7826f3..916c55d6ce9 100644
> >> >> > --- a/builtin/pack-objects.c
> >> >> > +++ b/builtin/pack-objects.c
> >> >> > @@ -1204,11 +1204,13 @@ static void write_pack_file(void)
> >> >> > * If so, rewrite it like in fast-import
> >> >> > */
> >> >> > if (pack_to_stdout) {
> >> >> > - finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_CLOSE);
> >> >> > + finalize_hashfile(f, hash, FSYNC_COMPONENT_NONE,
> >> >> > + CSUM_HASH_IN_STREAM | CSUM_CLOSE);
> >> >>
> >> >> Not really related to this per-se, but since you're touching the API
> >> >> everything goes through I wonder if callers should just always try to
> >> >> fsync, and we can just catch EROFS and EINVAL in the wrapper if someone
> >> >> tries to flush stdout, or catch the fd at that lower level.
> >> >>
> >> >> Or maybe there's a good reason for this...
> >> >
> >> > It's platform dependent, but I'd expect fsync would do something for
> >> > pipes or stdout redirected to a file. In these cases we really don't
> >> > want to fsync since we have no idea what we're talking to and we're
> >> > potentially worsening performance for probably no benefit.
> >>
> >> Yeah maybe we should just leave it be.
> >>
> >> I'd think the C library returning EINVAL would be a trivial performance
> >> cost though.
> >>
> >> It just seemed odd to hardcode assumptions about what can and can't be
> >> synced when the POSIX defined function will also tell us that.
> >>
> >
> > Redirecting stdout to a file seems like a common usage for this
> > command. That would definitely be fsyncable, but Git has no idea what
> > its proper category is since there's no way to know the purpose or
> > lifetime of the packfile. I'm going to leave this be, because I'd
> > posit that "can it be fsynced?" is not the same as "should it be
> > fsynced?". The latter question can't be answered for stdout.
>
> As noted this was just an aside, and I don't even know if any OS would
> do anything meaningful with an fsync() of such a FD anyway.
>
The underlying fsync primitive does have a meaning on Windows for
pipes, but it's certainly not what Git would want to do. Also if
stdout is redirected to a file, I'm pretty sure that UNIX OSes would
respect the fsync call. However it's not meaningful in the sense of
the git repository, since we don't know what the packfile is or why it
was created.
> I just don't see why we wouldn't say:
>
> 1. We're syncing this category of thing
> 2. Try it
> 3. If fsync returns "can't fsync that sort of thing" move on
>
> As opposed to trying to shortcut #3 by doing the detection ourselves.
>
> I.e. maybe there was a good reason, but it seemed to be some easy
> potential win for more simplification, since you were re-doing and
> simplifying some of the interface anyway...
We're trying to be deliberate about what we're fsyncing. Fsyncing an
unknown file created by the packfile code doesn't move us in that
direction. In your taxonomy we don't know (1), "what is this category
of thing?" Sure it's got the packfile format, but is not known to be
an actual packfile that's part of the repository.
> >>
> >> >> > [...]
> >> >> > +/*
> >> >> > + * These values are used to help identify parts of a repository to fsync.
> >> >> > + * FSYNC_COMPONENT_NONE identifies data that will not be a persistent part of the
> >> >> > + * repository and so shouldn't be fsynced.
> >> >> > + */
> >> >> > +enum fsync_component {
> >> >> > + FSYNC_COMPONENT_NONE = 0,
> >> >>
> >> >> I haven't read ahead much but in most other such cases we don't define
> >> >> the "= 0", just start at 1<<0, then check the flags elsewhere...
> >> >>
> >> >> > +static const struct fsync_component_entry {
> >> >> > + const char *name;
> >> >> > + enum fsync_component component_bits;
> >> >> > +} fsync_component_table[] = {
> >> >> > + { "loose-object", FSYNC_COMPONENT_LOOSE_OBJECT },
> >> >> > + { "pack", FSYNC_COMPONENT_PACK },
> >> >> > + { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA },
> >> >> > + { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH },
> >> >> > + { "objects", FSYNC_COMPONENTS_OBJECTS },
> >> >> > + { "default", FSYNC_COMPONENTS_DEFAULT },
> >> >> > + { "all", FSYNC_COMPONENTS_ALL },
> >> >> > +};
> >> >> > +
> >> >> > +static enum fsync_component parse_fsync_components(const char *var, const char *string)
> >> >> > +{
> >> >> > + enum fsync_component output = 0;
> >> >> > +
> >> >> > + if (!strcmp(string, "none"))
> >> >> > + return output;
> >> >> > +
> >> >> > + while (string) {
> >> >> > + int i;
> >> >> > + size_t len;
> >> >> > + const char *ep;
> >> >> > + int negated = 0;
> >> >> > + int found = 0;
> >> >> > +
> >> >> > + string = string + strspn(string, ", \t\n\r");
> >> >>
> >> >> Aside from the "use a list" isn't this hardcoding some windows-specific
> >> >> assumptions with \n\r? Maybe not...
> >> >
> >> > I shamelessly stole this code from parse_whitespace_rule. I thought
> >> > about making a helper to be called by both functions, but the amount
> >> > of state going into and out of the wrapper via arguments was
> >> > substantial and seemed to negate the benefit of deduplication.
> >>
> >> FWIW string_list_split() is easier to work with in those cases, or at
> >> least I think so...
> >
> > This code runs at startup for a variable that may be present on some
> > installations. The nice property of the current patch's code is that
> > it's already a well-tested pattern that doesn't do any allocations as
> > it's working, unlike string_list_split().
>
> Multi-value config would also get you fewer allocations :)
>
> Anyway, I mainly meant to point out that for stuff like this it's fine
> to optimize it for ease rather than micro-optimize allocations. Those
> really aren't a bottleneck on this scale.
>
> Even in that case there's string_list_split_in_place(), which can be a
> bit nicer than manual C-string fiddling.
>
Am I allowed to change the config value string in place? The
core.whitespace code is careful not to modify the string. I kind of
like the parse_ws_error_highlight code a little better now that I've
seen it, but I think the current code is fine too.
> > I hope you know that I appreciate your review feedback, even though
> > I'm pushing back on most of it so far this round. I'll be sending v3
> > to the list soon after giving it another look over.
>
> Sure, no worries. Just hoping to help. If you go for something different
> etc. that's fine. Just hoping to bridge the gap in some knowledge /
> offer potentially interesting suggestions (some of which may be dead
> ends, like the config glob thing...).
Thanks,
Neeraj
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v3 0/4] A design for future-proofing fsync() configuration
2021-12-09 0:57 ` [PATCH v3 0/4] " Neeraj K. Singh via GitGitGadget
` (3 preceding siblings ...)
2021-12-09 0:57 ` [PATCH v3 4/4] core.fsync: add a `derived-metadata` aggregate option Neeraj Singh via GitGitGadget
@ 2022-01-08 1:13 ` Neeraj Singh
2022-01-09 0:55 ` rsbecker
2022-02-01 3:33 ` [PATCH v4 " Neeraj K. Singh via GitGitGadget
5 siblings, 1 reply; 122+ messages in thread
From: Neeraj Singh @ 2022-01-08 1:13 UTC (permalink / raw)
To: Neeraj K. Singh via GitGitGadget
Cc: Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren,
Ævar Arnfjörð Bjarmason, Patrick Steinhardt,
Neeraj K. Singh
Hello Everyone,
I wanted to revive this thread in the new year.
To summarize the current state of affairs:
* The current fsync patch series implements two new configuration options:
core.fsync = <comma-separate list> -- select which repo
components will be fsynced
core.fsyncMethod = fsync|writeout-only -- select what form of
fsyncing will be done
* This patch series now ignores core.fsyncObjectFiles with a
deprecation warning pointing the user at core.fsync.
* There is a follow-on series that will extend the core.fsyncMethod to
also include a `batch` mode that speeds up bulk operations by avoiding
repeated disk cache flushes.
* I developed the current mechanism after Ævar pointed out that the
original `core.fsyncObjectFiles=batch` change would cause older
versions of Git to die() when exposed to a new configuration. There
were also several fsync changes floating around, including Patrick
Steinhardts `core.fsyncRefFiles` change [1] and Eric Wong's
`core.fsync = false` change [2].
* The biggest sticking points are in [3]. The fundamental
disagreement is about whether core.fsync should look like:
A) core.fsync = objects,commit-graph [current patch implementation]
or
B) core.fsync = objects
core.fsync = commit-graph [Ævar's multivalued proposal].
I prefer sticking with (A) for reasons spelled out in the thread. I'm
happy to re-litigate this discussion though.
* There's also a sticking point about whether we should fsync when
invoking pack-objects against stdout. I think that mostly reflects a
missing comment in the code rather than a real disagreement.
* Now that ew/test-wo-fsync has been integrated, there's some
redundancy between core.fsync=none and Eric's patch.
Open questions:
1) What format should we use for the core.fsync configuration to
select individual repo components to sync?
2) Are we okay with deprecating core.fsyncObjectFiles in a single
release with a warning?
3) Is it reasonable to expect people adding new persistent files to
add and document new values of the core.fsync settings?
Thanks,
Neeraj
[1] https://lore.kernel.org/git/20211030103950.M489266@dcvr/
[2] https://lore.kernel.org/git/20211028002102.19384-1-e@80x24.org/
[3] https://lore.kernel.org/git/211207.86wnkgo9fv.gmgdl@evledraar.gmail.com/
^ permalink raw reply [flat|nested] 122+ messages in thread
* RE: [PATCH v3 0/4] A design for future-proofing fsync() configuration
2022-01-08 1:13 ` [PATCH v3 0/4] A design for future-proofing fsync() configuration Neeraj Singh
@ 2022-01-09 0:55 ` rsbecker
2022-01-10 19:00 ` Neeraj Singh
0 siblings, 1 reply; 122+ messages in thread
From: rsbecker @ 2022-01-09 0:55 UTC (permalink / raw)
To: 'Neeraj Singh',
'Neeraj K. Singh via GitGitGadget'
Cc: 'Git List', 'Bagas Sanjaya',
'Elijah Newren',
'Ævar Arnfjörð Bjarmason',
'Patrick Steinhardt', 'Neeraj K. Singh'
On January 7, 2022 8:14 PM, Neeraj Singh wrote:
> Hello Everyone,
> I wanted to revive this thread in the new year.
>
> To summarize the current state of affairs:
> * The current fsync patch series implements two new configuration options:
> core.fsync = <comma-separate list> -- select which repo components will
> be fsynced
> core.fsyncMethod = fsync|writeout-only -- select what form of fsyncing
> will be done
>
> * This patch series now ignores core.fsyncObjectFiles with a deprecation
> warning pointing the user at core.fsync.
>
> * There is a follow-on series that will extend the core.fsyncMethod to also
> include a `batch` mode that speeds up bulk operations by avoiding repeated
> disk cache flushes.
>
> * I developed the current mechanism after Ævar pointed out that the original
> `core.fsyncObjectFiles=batch` change would cause older versions of Git to
> die() when exposed to a new configuration. There were also several fsync
> changes floating around, including Patrick Steinhardts `core.fsyncRefFiles`
> change [1] and Eric Wong's `core.fsync = false` change [2].
>
> * The biggest sticking points are in [3]. The fundamental disagreement is
> about whether core.fsync should look like:
> A) core.fsync = objects,commit-graph [current patch implementation]
> or
> B) core.fsync = objects
> core.fsync = commit-graph [Ævar's multivalued proposal].
> I prefer sticking with (A) for reasons spelled out in the thread. I'm happy to re-
> litigate this discussion though.
>
> * There's also a sticking point about whether we should fsync when invoking
> pack-objects against stdout. I think that mostly reflects a missing comment in
> the code rather than a real disagreement.
>
> * Now that ew/test-wo-fsync has been integrated, there's some redundancy
> between core.fsync=none and Eric's patch.
>
> Open questions:
> 1) What format should we use for the core.fsync configuration to select
> individual repo components to sync?
> 2) Are we okay with deprecating core.fsyncObjectFiles in a single release with
> a warning?
> 3) Is it reasonable to expect people adding new persistent files to add and
> document new values of the core.fsync settings?
>
> Thanks,
> Neeraj
>
> [1] https://lore.kernel.org/git/20211030103950.M489266@dcvr/
> [2] https://lore.kernel.org/git/20211028002102.19384-1-e@80x24.org/
> [3]
> https://lore.kernel.org/git/211207.86wnkgo9fv.gmgdl@evledraar.gmail.com/
Neeraj,
Please remember that fsync() is operating system and version specific. You cannot make any assumptions about what is supported and what is not. I have recently had issues with git built on a recent operating system not running on a version from 2020. The proposed patches do not work, as I recall, in a portable manner, so caution is required making this change. You can expect this not to work on some platforms and some versions. Please account for that. Requiring users who are not aware of OS details to configure git to function at all is a bad move, in my view - which has not changed since last time.
Thanks,
Randall
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v3 0/4] A design for future-proofing fsync() configuration
2022-01-09 0:55 ` rsbecker
@ 2022-01-10 19:00 ` Neeraj Singh
0 siblings, 0 replies; 122+ messages in thread
From: Neeraj Singh @ 2022-01-10 19:00 UTC (permalink / raw)
To: Randall S. Becker
Cc: Neeraj K. Singh via GitGitGadget, Git List, Bagas Sanjaya,
Elijah Newren, Ævar Arnfjörð Bjarmason,
Patrick Steinhardt, Neeraj K. Singh
On Sat, Jan 8, 2022 at 4:55 PM <rsbecker@nexbridge.com> wrote:
>
> Please remember that fsync() is operating system and version specific. You cannot make any assumptions about what is supported and what is not. I have recently had issues with git built on a recent operating system not running on a version from 2020. The proposed patches do not work, as I recall, in a portable manner, so caution is required making this change. You can expect this not to work on some platforms and some versions. Please account for that. Requiring users who are not aware of OS details to configure git to function at all is a bad move, in my view - which has not changed since last time.
>
There was already an implied configuration of fsync in the Git
codebase. None of the defaults are changing--assuming that a user
does not explicitly configure the core.fsync setting, Git should work
the same as it always has. I don't believe the current patch series
introduces any new incompatibilities.
Thanks,
Neeraj
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v2 2/3] core.fsync: introduce granular fsync control
2021-12-09 6:18 ` Neeraj Singh
@ 2022-01-18 23:50 ` Neeraj Singh
2022-01-19 15:28 ` Ævar Arnfjörð Bjarmason
2022-01-19 14:52 ` Ævar Arnfjörð Bjarmason
1 sibling, 1 reply; 122+ messages in thread
From: Neeraj Singh @ 2022-01-18 23:50 UTC (permalink / raw)
To: Ævar Arnfjörð Bjarmason
Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker,
Bagas Sanjaya, Elijah Newren, Patrick Steinhardt, Neeraj Singh
Hi Ævar,
Could you please respond to the parent message?
To summarize the main points and questions:
1) The current patch series design of core.fsync favors ease of
modification of the complete configuration as a single atomic config
step, at the cost of making it somewhat harder to derive a new
configuration from an existing one. See [1] where this facility is
used.
2) Is there any existing configuration that uses the multi-value
schema you proposed? The diff.wsErrorHighlight setting is actually
comma-separated.
3) Is there any system you can point at or any support in the POSIX
spec for requiring fsync for anything other than durability of data
across system crash?
4) I think string_list_split_in_place is not valid for splitting a
comma-separated list in the config code, since the value coming from
the configset code is in some global data structure that might be used
again. It seems like we could have subtle problems down the line if we
mutate it.
[1] https://github.com/neerajsi-msft/git/commit/7e9749a7e94d26c88789459588997329c5130792#diff-ee0307657f5a76b723c8973db0dbd5a2ca62e14b02711b897418b35d78fc6023R1327
Thanks,
Neeraj
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v2 2/3] core.fsync: introduce granular fsync control
2021-12-09 6:18 ` Neeraj Singh
2022-01-18 23:50 ` Neeraj Singh
@ 2022-01-19 14:52 ` Ævar Arnfjörð Bjarmason
2022-01-28 1:28 ` Neeraj Singh
1 sibling, 1 reply; 122+ messages in thread
From: Ævar Arnfjörð Bjarmason @ 2022-01-19 14:52 UTC (permalink / raw)
To: Neeraj Singh
Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker,
Bagas Sanjaya, Elijah Newren, Patrick Steinhardt, Neeraj Singh
On Wed, Dec 08 2021, Neeraj Singh wrote:
[Sorry about the late reply, and thanks for the downthread prodding]
> On Wed, Dec 8, 2021 at 8:46 PM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
>>
>>
>> On Wed, Dec 08 2021, Neeraj Singh wrote:
>>
>> > On Wed, Dec 8, 2021 at 2:17 AM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
>> >>
>> >>
>> >> On Tue, Dec 07 2021, Neeraj Singh wrote:
>> >>
>> >> > On Tue, Dec 7, 2021 at 5:01 AM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
>> >> >>
>> >> >>
>> >> >> On Tue, Dec 07 2021, Neeraj Singh via GitGitGadget wrote:
>> >> >>
>> >> >> > From: Neeraj Singh <neerajsi@microsoft.com>
>> >> >> >
>> >> >> > This commit introduces the `core.fsync` configuration
>> >> >> > knob which can be used to control how components of the
>> >> >> > repository are made durable on disk.
>> >> >> >
>> >> >> > This setting allows future extensibility of the list of
>> >> >> > syncable components:
>> >> >> > * We issue a warning rather than an error for unrecognized
>> >> >> > components, so new configs can be used with old Git versions.
>> >> >>
>> >> >> Looks good!
>> >> >>
>> >> >> > * We support negation, so users can choose one of the default
>> >> >> > aggregate options and then remove components that they don't
>> >> >> > want. The user would then harden any new components added in
>> >> >> > a Git version update.
>> >> >>
>> >> >> I think this config schema makes sense, but just a (I think important)
>> >> >> comment on the "how" not "what" of it. It's really much better to define
>> >> >> config as:
>> >> >>
>> >> >> [some]
>> >> >> key = value
>> >> >> key = value2
>> >> >>
>> >> >> Than:
>> >> >>
>> >> >> [some]
>> >> >> key = value,value2
>> >> >>
>> >> >> The reason is that "git config" has good support for working with
>> >> >> multi-valued stuff, so you can do e.g.:
>> >> >>
>> >> >> git config --get-all -z some.key
>> >> >>
>> >> >> And you can easily (re)set such config e.g. with --replace-all etc., but
>> >> >> for comma-delimited you (and users) need to do all that work themselves.
>> >> >>
>> >> >> Similarly instead of:
>> >> >>
>> >> >> some.key = want-this
>> >> >> some.key = -not-this
>> >> >> some.key = but-want-this
>> >> >>
>> >> >> I think it's better to just have two lists, one inclusive another
>> >> >> exclusive. E.g. see "log.decorate" and "log.excludeDecoration",
>> >> >> "transfer.hideRefs"
>> >> >>
>> >> >> Which would mean:
>> >> >>
>> >> >> core.fsync = want-this
>> >> >> core.fsyncExcludes = -not-this
>> >> >>
>> >> >> For some value of "fsyncExcludes", maybe "noFsync"? Anyway, just a
>> >> >> suggestion on making this easier for users & the implementation.
>> >> >>
>> >> >
>> >> > Maybe there's some way to handle this I'm unaware of, but a
>> >> > disadvantage of your multi-valued config proposal is that it's harder,
>> >> > for example, for a per-repo config store to reasonably override a
>> >> > per-user config store. With the configuration scheme as-is, I can
>> >> > have a per-user setting like `core.fsync=all` which covers my typical
>> >> > repos, but then have a maintainer repo with a private setting of
>> >> > `core.fsync=none` to speed up cases where I'm mostly working with
>> >> > other people's changes that are backed up in email or server-side
>> >> > repos. The latter setting conveniently overrides the former setting
>> >> > in all aspects.
>> >>
>> >> Even if you turn just your comma-delimited proposal into a list proposal
>> >> can't we just say that the last one wins? Then it won't matter what cmae
>> >> before, you'd specify "core.fsync=none" in your local .git/config.
>> >>
>> >> But this is also a general issue with a bunch of things in git's config
>> >> space. I'd rather see us use the list-based values and just come up with
>> >> some general way to reset them that works for all keys, rather than
>> >> regretting having comma-delimited values that'll be harder to work with
>> >> & parse, which will be a legacy wart if/when we come up with a way to
>> >> say "reset all previous settings".
>> >>
>> >> > Also, with the core.fsync and core.fsyncExcludes, how would you spell
>> >> > "don't sync anything"? Would you still have the aggregate options.?
>> >>
>> >> core.fsyncExcludes = *
>> >>
>> >> I.e. the same as the core.fsync=none above, anyway I still like the
>> >> wildcard thing below a bit more...
>> >
>> > I'm not going to take this feedback unless there are additional votes
>> > from the Git community in this direction. I make the claim that
>> > single-valued comma-separated config lists are easier to work with in
>> > the existing Git infrastructure.
>>
>> Easier in what sense? I showed examples of how "git-config" trivially
>> works with multi-valued config, but for comma-delimited you'll need to
>> write your own shellscript boilerplate around simple things like adding
>> values, removing existing ones etc.
>>
>> I.e. you can use --add, --unset, --unset-all, --get, --get-all etc.
>>
>
> I see what you're saying for cases where someone would want to set a
> core.fsync setting that's derived from the user's current config. But
> I'm guessing that the dominant use case is someone setting a new fsync
> configuration that achieves some atomic goal with respect to a given
> repository. Like "this is a throwaway, so sync nothing" or "this is
> really important, so sync all objects and refs and the index".
Whether it's multi-value or comma-separated you could do:
-c core.fsync=[none|false]
To sync nothing, i.e. if we see "none/false" it doesn't matter if we saw
core.fsync=loose-object or whatever before, it would override it, ditto
for "all".
>> > parsing code for the core.whitespace variable and users are used to
>> > this syntax there. There are several other comma-separated lists in
>> > the config space, so this construct has precedence and will be with
>> > Git for some time.
>>
>> That's not really an argument either way for why we'd pick X over Y for
>> something new. We've got some comma-delimited, some multi-value (I'm
>> fairly sure we have more multi-value).
>>
>
> My main point here is that there's precedent for patch's current exact
> schema for a config in the same core config leaf of the Documentation.
> It seems from your comments that we'd have to invent and document some
> new convention for "reset" of a multi-valued config. So you're
> suggesting that I solve an extra set of problems to get this change
> in. Just want to remind you that my personal itch to scratch is to
> get the underlying mechanism in so that git-for-windows can set its
> default setting to batch mode. I'm not expecting many users to
> actually configure this setting to any non-default value.
Me neither. I think people will most likely set this once in
~/.gitconfig or /etc/gitconfig.
We have some config keys that are multi-value and either comma-separated
or space-separated, e.g. core.alternateRefsPrefixes
Then we have e.g. blame.ignoreRevsFile which is multi-value, and further
has the convention that setting it to an empty value clears the
list. which would scratch the "override existing" itch.
format.notes, versionsort.suffix, transfer.hideRefs, branch.<name>.merge
are exmples of existing multi-value config.
>> > Also, fsync configurations aren't composable like
>> > some other configurations may be. It makes sense to have a holistic
>> > singular fsync configuration, which is best represented by a single
>> > variable.
>>
>> What's a "variable" here? We call these "keys", you can have a
>> single-value key like user.name that you get with --get, or a
>> multi-value key like say branch.<name>.merge or push.pushOption that
>> you'd get with --get-all.
>
> Yeah, I meant "key". I conflated the config key with the underlying
> global variable in git.
*nod*
>> I think you may be referring to either not wanting these to be
>> "inherited" (which is not really a think we do for anything else in
>> config), or lacking the ability to "reset".
>>
>> For the latter if that's absolutely needed we could just use the same
>> trick as "diff.wsErrorHighlight" uses of making an empty value "reset"
>> the list, and you'd get better "git config" support for editing it.
>>
>
> My reading of the code is that diff.wsErrorHighlight is a comma
> separated list and not a multi-valued config. Actually I haven't yet
> found an existing multi-valued config (not sure how to grep for it).
Yes, I think I conflated it with one of the ones above when I wrote
this.
>> >> >> > This also supports the common request of doing absolutely no
>> >> >> > fysncing with the `core.fsync=none` value, which is expected
>> >> >> > to make the test suite faster.
>> >> >>
>> >> >> Let's just use the git_parse_maybe_bool() or git_parse_maybe_bool_text()
>> >> >> so we'll accept "false", "off", "no" like most other such config?
>> >> >
>> >> > Junio's previous feedback when discussing batch mode [1] was to offer
>> >> > less flexibility when parsing new values of these configuration
>> >> > options. I agree with his statement that "making machine-readable
>> >> > tokens be spelled in different ways is a 'disease'." I'd like to
>> >> > leave this as-is so that the documentation can clearly state the exact
>> >> > set of allowable values.
>> >> >
>> >> > [1] https://lore.kernel.org/git/xmqqr1dqzyl7.fsf@gitster.g/
>> >>
>> >> I think he's talking about batch, Batch, BATCH, bAtCh etc. there. But
>> >> the "maybe bool" is a stanard pattern we use.
>> >>
>> >> I don't think we'd call one of these 0, off, no or false etc. to avoid
>> >> confusion, so then you can use git_parse_maybe_...()
>> >
>> > I don't see the advantage of having multiple ways of specifying
>> > "none". The user can read the doc and know exactly what to write. If
>> > they write something unallowable, they get a clear warning and they
>> > can read the doc again to figure out what to write. This isn't a
>> > boolean options at all, so why should we entertain bool-like ways of
>> > spelling it?
>>
>> It's not boolean, it's multi-value and one of the values includes a true
>> or false boolean value. You just spell it "none".
>>
>> I think both this and your comment above suggest that you think there's
>> no point in this because you haven't interacted with/used "git config"
>> as a command line or API mechanism, but have just hand-crafted config
>> files.
>>
>> That's fair enough, but there's a lot of tooling that benefits from the
>> latter.
>
> My batch mode perf tests (on github, not yet submitted to the list)
> use `git -c core.fsync=<foo>` to set up a per-process config. I
> haven't used the `git config` writing support in a while, so I haven't
> deeply thought about that. However, I favor simplifying the use case
> of "atomically" setting a new holistic core.fsync value versus the use
> case of deriving a new core.fsync value from the preexisting value.
If you implement it like blame.ignoreRevsFile you can have your cake and
eat it too, i.e.:
-c core.fsync= core.fsync=loose-object
ensures only loose objects are synced, as with your single-value config,
but I'd think what you'd be more likely to actually mean would be:
# With "core.fsync=pack" set in ~/.gitconfig
-c core.fsync=loose-object
I.e. that the common case is "I want this to be synced here", not that
you'd like to declare sync policy from scratch every time.
In any case, on this general topic my main point is that the
git-config(1) command has pretty integration for multi-value if you do
it that way, and not for comma-delimited. I.e. you get --add, --unset,
--unset-all, --get, --get-all etc.
So I think for anything new it makes sense to lean into that, I think
most of these existing comma-delimited ones are ones we'd do differently
today on reflection.
And if you suppor the "empty resets" like blame.ignoreRevsFile it seems
to me you'll have your cake & eat it too.
>> E.g.:
>>
>> $ git -c core.fsync=off config --type=bool core.fsync
>> false
>> $ git -c core.fsync=blah config --type=bool core.fsync
>> fatal: bad boolean config value 'blah' for 'core.fsync'
>>
>> Here we can get 'git config' to normalize what you call 'none', and you
>> can tell via exit codes/normalization if it's "false". But if you invent
>> a new term for "false" you can't do that as easily.
>>
>> We have various historical keys that take odd exceptions to that,
>> e.g. "never", but unless we have a good reason to let's not invent more
>> exceptions.
>>
>> >> >> > Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
>> >> >> > ---
>> >> >> > Documentation/config/core.txt | 27 +++++++++----
>> >> >> > builtin/fast-import.c | 2 +-
>> >> >> > builtin/index-pack.c | 4 +-
>> >> >> > builtin/pack-objects.c | 8 ++--
>> >> >> > bulk-checkin.c | 5 ++-
>> >> >> > cache.h | 39 +++++++++++++++++-
>> >> >> > commit-graph.c | 3 +-
>> >> >> > config.c | 76 ++++++++++++++++++++++++++++++++++-
>> >> >> > csum-file.c | 5 ++-
>> >> >> > csum-file.h | 3 +-
>> >> >> > environment.c | 1 +
>> >> >> > midx.c | 3 +-
>> >> >> > object-file.c | 3 +-
>> >> >> > pack-bitmap-write.c | 3 +-
>> >> >> > pack-write.c | 13 +++---
>> >> >> > read-cache.c | 2 +-
>> >> >> > 16 files changed, 164 insertions(+), 33 deletions(-)
>> >> >> >
>> >> >> > diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
>> >> >> > index dbb134f7136..4f1747ec871 100644
>> >> >> > --- a/Documentation/config/core.txt
>> >> >> > +++ b/Documentation/config/core.txt
>> >> >> > @@ -547,6 +547,25 @@ core.whitespace::
>> >> >> > is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent`
>> >> >> > errors. The default tab width is 8. Allowed values are 1 to 63.
>> >> >> >
>> >> >> > +core.fsync::
>> >> >> > + A comma-separated list of parts of the repository which should be
>> >> >> > + hardened via the core.fsyncMethod when created or modified. You can
>> >> >> > + disable hardening of any component by prefixing it with a '-'. Later
>> >> >> > + items take precedence over earlier ones in the list. For example,
>> >> >> > + `core.fsync=all,-pack-metadata` means "harden everything except pack
>> >> >> > + metadata." Items that are not hardened may be lost in the event of an
>> >> >> > + unclean system shutdown.
>> >> >> > ++
>> >> >> > +* `none` disables fsync completely. This must be specified alone.
>> >> >> > +* `loose-object` hardens objects added to the repo in loose-object form.
>> >> >> > +* `pack` hardens objects added to the repo in packfile form.
>> >> >> > +* `pack-metadata` hardens packfile bitmaps and indexes.
>> >> >> > +* `commit-graph` hardens the commit graph file.
>> >> >> > +* `objects` is an aggregate option that includes `loose-objects`, `pack`,
>> >> >> > + `pack-metadata`, and `commit-graph`.
>> >> >> > +* `default` is an aggregate option that is equivalent to `objects,-loose-object`
>> >> >> > +* `all` is an aggregate option that syncs all individual components above.
>> >> >> > +
>> >> >>
>> >> >> It's probably a *bit* more work to set up, but I wonder if this wouldn't
>> >> >> be simpler if we just said (and this is partially going against what I
>> >> >> noted above):
>> >> >>
>> >> >> == BEGIN DOC
>> >> >>
>> >> >> core.fsync is a multi-value config variable where each item is a
>> >> >> pathspec that'll get matched the same way as 'git-ls-files' et al.
>> >> >>
>> >> >> When we sync pretend that a path like .git/objects/de/adbeef... is
>> >> >> relative to the top-level of the git
>> >> >> directory. E.g. "objects/de/adbeaf.." or "objects/pack/...".
>> >> >>
>> >> >> You can then supply a list of wildcards and exclusions to configure
>> >> >> syncing. or "false", "off" etc. to turn it off. These are synonymous
>> >> >> with:
>> >> >>
>> >> >> ; same as "false"
>> >> >> core.fsync = ":!*"
>> >> >>
>> >> >> Or:
>> >> >>
>> >> >> ; same as "true"
>> >> >> core.fsync = "*"
>> >> >>
>> >> >> Or, to selectively sync some things and not others:
>> >> >>
>> >> >> ;; Sync objects, but not "info"
>> >> >> core.fsync = ":!objects/info/**"
>> >> >> core.fsync = "objects/**"
>> >> >>
>> >> >> See gitrepository-layout(5) for details about what sort of paths you
>> >> >> might be expected to match. Not all paths listed there will go through
>> >> >> this mechanism (e.g. currently objects do, but nothing to do with config
>> >> >> does).
>> >> >>
>> >> >> We can and will match this against "fake paths", e.g. when writing out
>> >> >> packs we may match against just the string "objects/pack", we're not
>> >> >> going to re-check if every packfile we're writing matches your globs,
>> >> >> ditto for loose objects. Be reasonable!
>> >> >>
>> >> >> This metharism is intended as a shorthand that provides some flexibility
>> >> >> when fsyncing, while not forcing git to come up with labels for all
>> >> >> paths the git dir, or to support crazyness like "objects/de/adbeef*"
>> >> >>
>> >> >> More paths may be added or removed in the future, and we make no
>> >> >> promises that we won't move things around, so if in doubt use
>> >> >> e.g. "true" or a wide pattern match like "objects/**". When in doubt
>> >> >> stick to the golden path of examples provided in this documentation.
>> >> >>
>> >> >> == END DOC
>> >> >>
>> >> >>
>> >> >> It's a tad more complex to set up, but I wonder if that isn't worth
>> >> >> it. It nicely gets around any current and future issues of deciding what
>> >> >> labels such as "loose-object" etc. to pick, as well as slotting into an
>> >> >> existing method of doing exclude/include lists.
>> >> >>
>> >> >
>> >> > I think this proposal is a lot of complexity to avoid coming up with a
>> >> > new name for syncable things as they are added to Git. A path based
>> >> > mechanism makes it hard to document for the (advanced) user what the
>> >> > full set of things is and how it might change from release to release.
>> >> > I think the current core.fsync scheme is a bit easier to understand,
>> >> > query, and extend.
>> >>
>> >> We document it in gitrepository-layout(5). Yeah it has some
>> >> disadvantages, but one advantage is that you could make the
>> >> composability easy. I.e. if last exclude wins then a setting of:
>> >>
>> >> core.fsync = ":!*"
>> >> core.fsync = "objects/**"
>> >>
>> >> Would reset all previous matches & only match objects/**.
>> >>
>> >
>> > The value of changing this is predicated on taking your previous
>> > multi-valued config proposal, which I'm still not at all convinced
>> > about.
>>
>> They're orthagonal. I.e. you get benefits from multi-value with or
>> without this globbing mechanism.
>>
>> In any case, I don't feel strongly about/am really advocating this
>> globbing mechanism. I just wondered if it wouldn't make things simpler
>> since it would sidestep the need to create any sort of categories for
>> subsets of gitrepository-layout(5), but maybe not...
>>
>> > The schema in the current (v1-v2) version of the patch already
>> > includes an example of extending the list of syncable things, and
>> > Patrick Steinhardt made it clear that he feels comfortable adding
>> > 'refs' to the same schema in a future change.
>> >
>> > I'll also emphasize that we're talking about a non-functional,
>> > relatively corner-case behavioral configuration. These values don't
>> > change how git's interface behaves except when the system crashes
>> > during a git command or shortly after one completes.
>>
>> That may be something some OS's promise, but it's not something fsync()
>> or POSIX promises. I.e. you might write a ref, but unless you fsync and
>> the relevant dir entries another process might not see the update, crash
>> or not.
>>
>
> I haven't seen any indication that POSIX requires an fsync for
> visiblity within a running system. I looked at the doc for open,
> write, and fsync, and saw no indication that it's posix compliant to
> require an fsync for visibility. I think an OS that required fsync
> for cross-process visiblity would fail to run Git for a myriad of
> other reasons and would likely lose all its users. I'm curious where
> you've seen documentation that allows such unhelpful behavior?
There's multiple unrelated and related things in this area. One is a
case where you'll e.g. write a file "foo" using stdio, spawn a program
to work on it in the same program, but it might not see it at all, or
see empty content, the latter being because you haven't flushed your I/O
buffers (which you can do via fsync()).
The former is that on *nix systems you're generally only guaranteed to
write to a fd, but not to have the associated metadata be synced for
you.
That is spelled out e.g. in the DESCRIPTION section of linux's fsync()
manpage: https://man7.org/linux/man-pages/man2/fdatasync.2.html
I don't know how much you follow non-Windows FS development, but there
was also a very well known "incident" early in ext4 where it leaned into
some permissive-by-POSIX behavior that caused data loss in practice on
buggy programs that didn't correctly use fsync() , since various tooling
had come to expect the stricter behavior of ext3:
https://lwn.net/Articles/328363/
That was explicitly in the area of fs metadata being discussed here.
Generally you can expect your VFS layer to be forgiving when it comes to
IPC, but even that is out the window when it comes to networked
filesystems, e.g. a shared git repository hosted on NFS.
>> That's an aside from these config design questions, and I think
>> most/(all?) OS's anyone cares about these days tend to make that
>> implicit promise as part of their VFS behavior, but we should probably
>> design such an interface to fsync() with such pedantic portability in
>> mind.
>
> Why? To be rude to such a hypothetical system, if a system were so
> insanely designed, it would be nuts to support it.
Because we know that right now the system calls we're invoking aren't
guaranteed to store data persistently to disk portably, although they do
so in practice on most modern OS's.
We're portably to a lot of platforms, and also need to keep e.g. NFS in
mind, so being able to ask for a pedantic mode when you care about data
retention at the cost of performance would be nice.
And because the fsync config mode you're proposing is thoroughly
non-standard, but is known to me much faster by leaning into known
attributes of specific FS's on specific OS's, if we're not running on
those it would be sensible to fall back to a stricter mode of
operation. E.g. syncing all 100 loose objects we just wrote, not just
the last one.
>> > While you may not personally love the proposed configuration
>> > interface, I'd want your view on some questions:
>> > 1. Is it easy for the (advanced) user to set a configuration?
>> > 2. Is it easy for the (advanced) user to see what was configured?
>> > 3. Is it easy for the Git community to build on this as we want to add
>> > things to the list of things to sync?
>> > a) Is there a good best practice configuration so that people can
>> > avoid losing integrity for new stuff that they are intending to sync.
>> > b) If someone has a custom configuration, can that custom
>> > configuration do something reasonable as they upgrade versions of Git?
>> > ** In response to this question, I might see some value
>> > in adding a 'derived-metadata' aggregate that can be disabled so that
>> > a custom configuration can exclude those as they change version to
>> > version.
>> > c) Is it too much maintenance overhead to consider how to present
>> > this configuration knob for any new hashfile or other datafile in the
>> > git repo?
>> > 4. Is there a good path forward to change the default syncable set,
>> > both in git-for-windows and in Git for other platforms?
>>
>> I'm not really sure this globbing this is a good idea, as noted above
>> just a suggestion etc.
>>
>> As noted there it just gets you out of the business of re-defining
>> gitrepository-layout(5), and assuming too much in advance about certain
>> use-cases.
>>
>> E.g. even "refs" might be too broad for some. I don't tend to be I/O
>> limited, but I could see how someone who would be would care about
>> refs/heads but not refs/remotes, or want to exclude logs/* but not the
>> refs updates themselves etc.
>
> This use-case is interesting (distinguishing remote refs from local
> refs). I think the difficulty of verifying (for even an advanced
> user) that the right fsyncing is actually happening still puts me on
> the side of having a carefully curated and documented set of syncable
> things rather than a file-path-based mechanism.
>
> Is this meaningful in the presumably nearby future world of the refsdb
> backend? Is that somehow split by remote versus local?
There is the upcoming "reftable" work, but that's probably 2-3 years out
at the earliest for series production workloads in git.git.
>> >> >> > diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
>> >> >> > index 857be7826f3..916c55d6ce9 100644
>> >> >> > --- a/builtin/pack-objects.c
>> >> >> > +++ b/builtin/pack-objects.c
>> >> >> > @@ -1204,11 +1204,13 @@ static void write_pack_file(void)
>> >> >> > * If so, rewrite it like in fast-import
>> >> >> > */
>> >> >> > if (pack_to_stdout) {
>> >> >> > - finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_CLOSE);
>> >> >> > + finalize_hashfile(f, hash, FSYNC_COMPONENT_NONE,
>> >> >> > + CSUM_HASH_IN_STREAM | CSUM_CLOSE);
>> >> >>
>> >> >> Not really related to this per-se, but since you're touching the API
>> >> >> everything goes through I wonder if callers should just always try to
>> >> >> fsync, and we can just catch EROFS and EINVAL in the wrapper if someone
>> >> >> tries to flush stdout, or catch the fd at that lower level.
>> >> >>
>> >> >> Or maybe there's a good reason for this...
>> >> >
>> >> > It's platform dependent, but I'd expect fsync would do something for
>> >> > pipes or stdout redirected to a file. In these cases we really don't
>> >> > want to fsync since we have no idea what we're talking to and we're
>> >> > potentially worsening performance for probably no benefit.
>> >>
>> >> Yeah maybe we should just leave it be.
>> >>
>> >> I'd think the C library returning EINVAL would be a trivial performance
>> >> cost though.
>> >>
>> >> It just seemed odd to hardcode assumptions about what can and can't be
>> >> synced when the POSIX defined function will also tell us that.
>> >>
>> >
>> > Redirecting stdout to a file seems like a common usage for this
>> > command. That would definitely be fsyncable, but Git has no idea what
>> > its proper category is since there's no way to know the purpose or
>> > lifetime of the packfile. I'm going to leave this be, because I'd
>> > posit that "can it be fsynced?" is not the same as "should it be
>> > fsynced?". The latter question can't be answered for stdout.
>>
>> As noted this was just an aside, and I don't even know if any OS would
>> do anything meaningful with an fsync() of such a FD anyway.
>>
>
> The underlying fsync primitive does have a meaning on Windows for
> pipes, but it's certainly not what Git would want to do. Also if
> stdout is redirected to a file, I'm pretty sure that UNIX OSes would
> respect the fsync call. However it's not meaningful in the sense of
> the git repository, since we don't know what the packfile is or why it
> was created.
I suggested that because I think it's probably nonsensical, but it's
nonsense that POSIX seems to explicitly tell us that it'll handle
(probably by silently doing nothing). So in terms of our interface we
could lean into that and avoid our own special-casing.
>> I just don't see why we wouldn't say:
>>
>> 1. We're syncing this category of thing
>> 2. Try it
>> 3. If fsync returns "can't fsync that sort of thing" move on
>>
>> As opposed to trying to shortcut #3 by doing the detection ourselves.
>>
>> I.e. maybe there was a good reason, but it seemed to be some easy
>> potential win for more simplification, since you were re-doing and
>> simplifying some of the interface anyway...
>
> We're trying to be deliberate about what we're fsyncing. Fsyncing an
> unknown file created by the packfile code doesn't move us in that
> direction. In your taxonomy we don't know (1), "what is this category
> of thing?" Sure it's got the packfile format, but is not known to be
> an actual packfile that's part of the repository.
We know it's a fd, isn't that sufficient? In any case, I'm fine with
also keeping it as is, I don't mean to split hairs here.
It just stuck out as an odd part of the interface, why treat some fd's
specially, instead of just throwing it all at the OS. Presumably the
first thing the OS will do is figure out if it's a syncable fd or not,
and act appropriately.
>> >>
>> >> >> > [...]
>> >> >> > +/*
>> >> >> > + * These values are used to help identify parts of a repository to fsync.
>> >> >> > + * FSYNC_COMPONENT_NONE identifies data that will not be a persistent part of the
>> >> >> > + * repository and so shouldn't be fsynced.
>> >> >> > + */
>> >> >> > +enum fsync_component {
>> >> >> > + FSYNC_COMPONENT_NONE = 0,
>> >> >>
>> >> >> I haven't read ahead much but in most other such cases we don't define
>> >> >> the "= 0", just start at 1<<0, then check the flags elsewhere...
>> >> >>
>> >> >> > +static const struct fsync_component_entry {
>> >> >> > + const char *name;
>> >> >> > + enum fsync_component component_bits;
>> >> >> > +} fsync_component_table[] = {
>> >> >> > + { "loose-object", FSYNC_COMPONENT_LOOSE_OBJECT },
>> >> >> > + { "pack", FSYNC_COMPONENT_PACK },
>> >> >> > + { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA },
>> >> >> > + { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH },
>> >> >> > + { "objects", FSYNC_COMPONENTS_OBJECTS },
>> >> >> > + { "default", FSYNC_COMPONENTS_DEFAULT },
>> >> >> > + { "all", FSYNC_COMPONENTS_ALL },
>> >> >> > +};
>> >> >> > +
>> >> >> > +static enum fsync_component parse_fsync_components(const char *var, const char *string)
>> >> >> > +{
>> >> >> > + enum fsync_component output = 0;
>> >> >> > +
>> >> >> > + if (!strcmp(string, "none"))
>> >> >> > + return output;
>> >> >> > +
>> >> >> > + while (string) {
>> >> >> > + int i;
>> >> >> > + size_t len;
>> >> >> > + const char *ep;
>> >> >> > + int negated = 0;
>> >> >> > + int found = 0;
>> >> >> > +
>> >> >> > + string = string + strspn(string, ", \t\n\r");
>> >> >>
>> >> >> Aside from the "use a list" isn't this hardcoding some windows-specific
>> >> >> assumptions with \n\r? Maybe not...
>> >> >
>> >> > I shamelessly stole this code from parse_whitespace_rule. I thought
>> >> > about making a helper to be called by both functions, but the amount
>> >> > of state going into and out of the wrapper via arguments was
>> >> > substantial and seemed to negate the benefit of deduplication.
>> >>
>> >> FWIW string_list_split() is easier to work with in those cases, or at
>> >> least I think so...
>> >
>> > This code runs at startup for a variable that may be present on some
>> > installations. The nice property of the current patch's code is that
>> > it's already a well-tested pattern that doesn't do any allocations as
>> > it's working, unlike string_list_split().
>>
>> Multi-value config would also get you fewer allocations :)
>>
>> Anyway, I mainly meant to point out that for stuff like this it's fine
>> to optimize it for ease rather than micro-optimize allocations. Those
>> really aren't a bottleneck on this scale.
>>
>> Even in that case there's string_list_split_in_place(), which can be a
>> bit nicer than manual C-string fiddling.
>>
>
> Am I allowed to change the config value string in place? The
> core.whitespace code is careful not to modify the string. I kind of
> like the parse_ws_error_highlight code a little better now that I've
> seen it, but I think the current code is fine too.
I don't remember offhand if that's safe, probably not. So you'll need a
copy here.
>> > I hope you know that I appreciate your review feedback, even though
>> > I'm pushing back on most of it so far this round. I'll be sending v3
>> > to the list soon after giving it another look over.
>>
>> Sure, no worries. Just hoping to help. If you go for something different
>> etc. that's fine. Just hoping to bridge the gap in some knowledge /
>> offer potentially interesting suggestions (some of which may be dead
>> ends, like the config glob thing...).
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v2 2/3] core.fsync: introduce granular fsync control
2022-01-18 23:50 ` Neeraj Singh
@ 2022-01-19 15:28 ` Ævar Arnfjörð Bjarmason
0 siblings, 0 replies; 122+ messages in thread
From: Ævar Arnfjörð Bjarmason @ 2022-01-19 15:28 UTC (permalink / raw)
To: Neeraj Singh
Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker,
Bagas Sanjaya, Elijah Newren, Patrick Steinhardt, Neeraj Singh
On Tue, Jan 18 2022, Neeraj Singh wrote:
> Hi Ævar,
> Could you please respond to the parent message?
I did just now in
https://lore.kernel.org/git/220119.8635ljoidt.gmgdl@evledraar.gmail.com/;
sorry about the delay. This thread fell off my radar.
> To summarize the main points and questions:
> 1) The current patch series design of core.fsync favors ease of
> modification of the complete configuration as a single atomic config
> step, at the cost of making it somewhat harder to derive a new
> configuration from an existing one. See [1] where this facility is
> used.
>
> 2) Is there any existing configuration that uses the multi-value
> schema you proposed? The diff.wsErrorHighlight setting is actually
> comma-separated.
I replied to that. To add a bit to that the comma-delimited thing isn't
any sort of a "blocker" or whatever in my mind.
I just wanted to point out that you could get the same with multi-value
with some better integration, and we might have our cake & eat it too.
But at the end of the day if you disagree you're doing the work, and
that's ultimately bikeshedding of the config interface. I'm fine with it
either way.
> 3) Is there any system you can point at or any support in the POSIX
> spec for requiring fsync for anything other than durability of data
> across system crash?
Some examples in the linked...
> 4) I think string_list_split_in_place is not valid for splitting a
> comma-separated list in the config code, since the value coming from
> the configset code is in some global data structure that might be used
> again. It seems like we could have subtle problems down the line if we
> mutate it.
Replied to that too, hope it's all useful. Thanks again for working on
this, it's really nice to have someone move the state of data integrity
in git forward.
> [1] https://github.com/neerajsi-msft/git/commit/7e9749a7e94d26c88789459588997329c5130792#diff-ee0307657f5a76b723c8973db0dbd5a2ca62e14b02711b897418b35d78fc6023R1327
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v2 2/3] core.fsync: introduce granular fsync control
2022-01-19 14:52 ` Ævar Arnfjörð Bjarmason
@ 2022-01-28 1:28 ` Neeraj Singh
0 siblings, 0 replies; 122+ messages in thread
From: Neeraj Singh @ 2022-01-28 1:28 UTC (permalink / raw)
To: Ævar Arnfjörð Bjarmason
Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker,
Bagas Sanjaya, Elijah Newren, Patrick Steinhardt, Neeraj Singh
On Wed, Jan 19, 2022 at 10:27 AM Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
>
>
> On Wed, Dec 08 2021, Neeraj Singh wrote:
>
> [Sorry about the late reply, and thanks for the downthread prodding]
I also apologize for the late reply here. I've been dealing with a
hospitalized parent this week.
>
> > On Wed, Dec 8, 2021 at 8:46 PM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
> >>
> >>
> >> On Wed, Dec 08 2021, Neeraj Singh wrote:
> >>
> >> > On Wed, Dec 8, 2021 at 2:17 AM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
> >> >>
> >> >>
> >> >> On Tue, Dec 07 2021, Neeraj Singh wrote:
> >> >>
> >> >> > On Tue, Dec 7, 2021 at 5:01 AM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
> >> >> >>
> >> >> >>
> >> >> >> On Tue, Dec 07 2021, Neeraj Singh via GitGitGadget wrote:
> >> >> >>
> >> >> >> > From: Neeraj Singh <neerajsi@microsoft.com>
> >> >> >> >
> >> >> >> > This commit introduces the `core.fsync` configuration
> >> >> >> > knob which can be used to control how components of the
> >> >> >> > repository are made durable on disk.
> >> >> >> >
> >> >> >> > This setting allows future extensibility of the list of
> >> >> >> > syncable components:
> >> >> >> > * We issue a warning rather than an error for unrecognized
> >> >> >> > components, so new configs can be used with old Git versions.
> >> >> >>
> >> >> >> Looks good!
> >> >> >>
> >> >> >> > * We support negation, so users can choose one of the default
> >> >> >> > aggregate options and then remove components that they don't
> >> >> >> > want. The user would then harden any new components added in
> >> >> >> > a Git version update.
> >> >> >>
> >> >> >> I think this config schema makes sense, but just a (I think important)
> >> >> >> comment on the "how" not "what" of it. It's really much better to define
> >> >> >> config as:
> >> >> >>
> >> >> >> [some]
> >> >> >> key = value
> >> >> >> key = value2
> >> >> >>
> >> >> >> Than:
> >> >> >>
> >> >> >> [some]
> >> >> >> key = value,value2
> >> >> >>
> >> >> >> The reason is that "git config" has good support for working with
> >> >> >> multi-valued stuff, so you can do e.g.:
> >> >> >>
> >> >> >> git config --get-all -z some.key
> >> >> >>
> >> >> >> And you can easily (re)set such config e.g. with --replace-all etc., but
> >> >> >> for comma-delimited you (and users) need to do all that work themselves.
> >> >> >>
> >> >> >> Similarly instead of:
> >> >> >>
> >> >> >> some.key = want-this
> >> >> >> some.key = -not-this
> >> >> >> some.key = but-want-this
> >> >> >>
> >> >> >> I think it's better to just have two lists, one inclusive another
> >> >> >> exclusive. E.g. see "log.decorate" and "log.excludeDecoration",
> >> >> >> "transfer.hideRefs"
> >> >> >>
> >> >> >> Which would mean:
> >> >> >>
> >> >> >> core.fsync = want-this
> >> >> >> core.fsyncExcludes = -not-this
> >> >> >>
> >> >> >> For some value of "fsyncExcludes", maybe "noFsync"? Anyway, just a
> >> >> >> suggestion on making this easier for users & the implementation.
> >> >> >>
> >> >> >
> >> >> > Maybe there's some way to handle this I'm unaware of, but a
> >> >> > disadvantage of your multi-valued config proposal is that it's harder,
> >> >> > for example, for a per-repo config store to reasonably override a
> >> >> > per-user config store. With the configuration scheme as-is, I can
> >> >> > have a per-user setting like `core.fsync=all` which covers my typical
> >> >> > repos, but then have a maintainer repo with a private setting of
> >> >> > `core.fsync=none` to speed up cases where I'm mostly working with
> >> >> > other people's changes that are backed up in email or server-side
> >> >> > repos. The latter setting conveniently overrides the former setting
> >> >> > in all aspects.
> >> >>
> >> >> Even if you turn just your comma-delimited proposal into a list proposal
> >> >> can't we just say that the last one wins? Then it won't matter what cmae
> >> >> before, you'd specify "core.fsync=none" in your local .git/config.
> >> >>
> >> >> But this is also a general issue with a bunch of things in git's config
> >> >> space. I'd rather see us use the list-based values and just come up with
> >> >> some general way to reset them that works for all keys, rather than
> >> >> regretting having comma-delimited values that'll be harder to work with
> >> >> & parse, which will be a legacy wart if/when we come up with a way to
> >> >> say "reset all previous settings".
> >> >>
> >> >> > Also, with the core.fsync and core.fsyncExcludes, how would you spell
> >> >> > "don't sync anything"? Would you still have the aggregate options.?
> >> >>
> >> >> core.fsyncExcludes = *
> >> >>
> >> >> I.e. the same as the core.fsync=none above, anyway I still like the
> >> >> wildcard thing below a bit more...
> >> >
> >> > I'm not going to take this feedback unless there are additional votes
> >> > from the Git community in this direction. I make the claim that
> >> > single-valued comma-separated config lists are easier to work with in
> >> > the existing Git infrastructure.
> >>
> >> Easier in what sense? I showed examples of how "git-config" trivially
> >> works with multi-valued config, but for comma-delimited you'll need to
> >> write your own shellscript boilerplate around simple things like adding
> >> values, removing existing ones etc.
> >>
> >> I.e. you can use --add, --unset, --unset-all, --get, --get-all etc.
> >>
> >
> > I see what you're saying for cases where someone would want to set a
> > core.fsync setting that's derived from the user's current config. But
> > I'm guessing that the dominant use case is someone setting a new fsync
> > configuration that achieves some atomic goal with respect to a given
> > repository. Like "this is a throwaway, so sync nothing" or "this is
> > really important, so sync all objects and refs and the index".
>
> Whether it's multi-value or comma-separated you could do:
>
> -c core.fsync=[none|false]
>
> To sync nothing, i.e. if we see "none/false" it doesn't matter if we saw
> core.fsync=loose-object or whatever before, it would override it, ditto
> for "all".
>
> >> > parsing code for the core.whitespace variable and users are used to
> >> > this syntax there. There are several other comma-separated lists in
> >> > the config space, so this construct has precedence and will be with
> >> > Git for some time.
> >>
> >> That's not really an argument either way for why we'd pick X over Y for
> >> something new. We've got some comma-delimited, some multi-value (I'm
> >> fairly sure we have more multi-value).
> >>
> >
> > My main point here is that there's precedent for patch's current exact
> > schema for a config in the same core config leaf of the Documentation.
> > It seems from your comments that we'd have to invent and document some
> > new convention for "reset" of a multi-valued config. So you're
> > suggesting that I solve an extra set of problems to get this change
> > in. Just want to remind you that my personal itch to scratch is to
> > get the underlying mechanism in so that git-for-windows can set its
> > default setting to batch mode. I'm not expecting many users to
> > actually configure this setting to any non-default value.
>
> Me neither. I think people will most likely set this once in
> ~/.gitconfig or /etc/gitconfig.
>
> We have some config keys that are multi-value and either comma-separated
> or space-separated, e.g. core.alternateRefsPrefixes
>
> Then we have e.g. blame.ignoreRevsFile which is multi-value, and further
> has the convention that setting it to an empty value clears the
> list. which would scratch the "override existing" itch.
>
> format.notes, versionsort.suffix, transfer.hideRefs, branch.<name>.merge
> are exmples of existing multi-value config.
>
Thanks for the examples. I can see the benefit of mutli-value for
most of those settings, but for versionsort.suffix, I'd personally
have wanted a comma-separated list.
> >> > Also, fsync configurations aren't composable like
> >> > some other configurations may be. It makes sense to have a holistic
> >> > singular fsync configuration, which is best represented by a single
> >> > variable.
> >>
> >> What's a "variable" here? We call these "keys", you can have a
> >> single-value key like user.name that you get with --get, or a
> >> multi-value key like say branch.<name>.merge or push.pushOption that
> >> you'd get with --get-all.
> >
> > Yeah, I meant "key". I conflated the config key with the underlying
> > global variable in git.
>
> *nod*
>
> >> I think you may be referring to either not wanting these to be
> >> "inherited" (which is not really a think we do for anything else in
> >> config), or lacking the ability to "reset".
> >>
> >> For the latter if that's absolutely needed we could just use the same
> >> trick as "diff.wsErrorHighlight" uses of making an empty value "reset"
> >> the list, and you'd get better "git config" support for editing it.
> >>
> >
> > My reading of the code is that diff.wsErrorHighlight is a comma
> > separated list and not a multi-valued config. Actually I haven't yet
> > found an existing multi-valued config (not sure how to grep for it).
>
> Yes, I think I conflated it with one of the ones above when I wrote
> this.
>
> >> >> >> > This also supports the common request of doing absolutely no
> >> >> >> > fysncing with the `core.fsync=none` value, which is expected
> >> >> >> > to make the test suite faster.
> >> >> >>
> >> >> >> Let's just use the git_parse_maybe_bool() or git_parse_maybe_bool_text()
> >> >> >> so we'll accept "false", "off", "no" like most other such config?
> >> >> >
> >> >> > Junio's previous feedback when discussing batch mode [1] was to offer
> >> >> > less flexibility when parsing new values of these configuration
> >> >> > options. I agree with his statement that "making machine-readable
> >> >> > tokens be spelled in different ways is a 'disease'." I'd like to
> >> >> > leave this as-is so that the documentation can clearly state the exact
> >> >> > set of allowable values.
> >> >> >
> >> >> > [1] https://lore.kernel.org/git/xmqqr1dqzyl7.fsf@gitster.g/
> >> >>
> >> >> I think he's talking about batch, Batch, BATCH, bAtCh etc. there. But
> >> >> the "maybe bool" is a stanard pattern we use.
> >> >>
> >> >> I don't think we'd call one of these 0, off, no or false etc. to avoid
> >> >> confusion, so then you can use git_parse_maybe_...()
> >> >
> >> > I don't see the advantage of having multiple ways of specifying
> >> > "none". The user can read the doc and know exactly what to write. If
> >> > they write something unallowable, they get a clear warning and they
> >> > can read the doc again to figure out what to write. This isn't a
> >> > boolean options at all, so why should we entertain bool-like ways of
> >> > spelling it?
> >>
> >> It's not boolean, it's multi-value and one of the values includes a true
> >> or false boolean value. You just spell it "none".
> >>
> >> I think both this and your comment above suggest that you think there's
> >> no point in this because you haven't interacted with/used "git config"
> >> as a command line or API mechanism, but have just hand-crafted config
> >> files.
> >>
> >> That's fair enough, but there's a lot of tooling that benefits from the
> >> latter.
> >
> > My batch mode perf tests (on github, not yet submitted to the list)
> > use `git -c core.fsync=<foo>` to set up a per-process config. I
> > haven't used the `git config` writing support in a while, so I haven't
> > deeply thought about that. However, I favor simplifying the use case
> > of "atomically" setting a new holistic core.fsync value versus the use
> > case of deriving a new core.fsync value from the preexisting value.
>
> If you implement it like blame.ignoreRevsFile you can have your cake and
> eat it too, i.e.:
>
> -c core.fsync= core.fsync=loose-object
>
> ensures only loose objects are synced, as with your single-value config,
> but I'd think what you'd be more likely to actually mean would be:
>
> # With "core.fsync=pack" set in ~/.gitconfig
> -c core.fsync=loose-object
>
> I.e. that the common case is "I want this to be synced here", not that
> you'd like to declare sync policy from scratch every time.
>
> In any case, on this general topic my main point is that the
> git-config(1) command has pretty integration for multi-value if you do
> it that way, and not for comma-delimited. I.e. you get --add, --unset,
> --unset-all, --get, --get-all etc.
>
> So I think for anything new it makes sense to lean into that, I think
> most of these existing comma-delimited ones are ones we'd do differently
> today on reflection.
>
> And if you suppor the "empty resets" like blame.ignoreRevsFile it seems
> to me you'll have your cake & eat it too.
>
I really don't like the multiple `-c core.fsync=` clauses. If I want
to do packs and loose-objects as a single config, I'd have to do:
* multi-value: `git -c core.fsync= -c core.fsync=pack -c
core.fsync=loose-object`
* comma-sep `git -c core.fsync=pack,loose-object`
Multi-valued configs are stateful and more verbose to configure.
Last-one-wins with comma-separated values has an advantage for
achieving a desired final configuration without regard for the
previous configuration, which is the way I expect the feature to be
used.
> >> E.g.:
> >>
> >> $ git -c core.fsync=off config --type=bool core.fsync
> >> false
> >> $ git -c core.fsync=blah config --type=bool core.fsync
> >> fatal: bad boolean config value 'blah' for 'core.fsync'
> >>
> >> Here we can get 'git config' to normalize what you call 'none', and you
> >> can tell via exit codes/normalization if it's "false". But if you invent
> >> a new term for "false" you can't do that as easily.
> >>
> >> We have various historical keys that take odd exceptions to that,
> >> e.g. "never", but unless we have a good reason to let's not invent more
> >> exceptions.
> >>
> >> >> >> > Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
> >> >> >> > ---
> >> >> >> > Documentation/config/core.txt | 27 +++++++++----
> >> >> >> > builtin/fast-import.c | 2 +-
> >> >> >> > builtin/index-pack.c | 4 +-
> >> >> >> > builtin/pack-objects.c | 8 ++--
> >> >> >> > bulk-checkin.c | 5 ++-
> >> >> >> > cache.h | 39 +++++++++++++++++-
> >> >> >> > commit-graph.c | 3 +-
> >> >> >> > config.c | 76 ++++++++++++++++++++++++++++++++++-
> >> >> >> > csum-file.c | 5 ++-
> >> >> >> > csum-file.h | 3 +-
> >> >> >> > environment.c | 1 +
> >> >> >> > midx.c | 3 +-
> >> >> >> > object-file.c | 3 +-
> >> >> >> > pack-bitmap-write.c | 3 +-
> >> >> >> > pack-write.c | 13 +++---
> >> >> >> > read-cache.c | 2 +-
> >> >> >> > 16 files changed, 164 insertions(+), 33 deletions(-)
> >> >> >> >
> >> >> >> > diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
> >> >> >> > index dbb134f7136..4f1747ec871 100644
> >> >> >> > --- a/Documentation/config/core.txt
> >> >> >> > +++ b/Documentation/config/core.txt
> >> >> >> > @@ -547,6 +547,25 @@ core.whitespace::
> >> >> >> > is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent`
> >> >> >> > errors. The default tab width is 8. Allowed values are 1 to 63.
> >> >> >> >
> >> >> >> > +core.fsync::
> >> >> >> > + A comma-separated list of parts of the repository which should be
> >> >> >> > + hardened via the core.fsyncMethod when created or modified. You can
> >> >> >> > + disable hardening of any component by prefixing it with a '-'. Later
> >> >> >> > + items take precedence over earlier ones in the list. For example,
> >> >> >> > + `core.fsync=all,-pack-metadata` means "harden everything except pack
> >> >> >> > + metadata." Items that are not hardened may be lost in the event of an
> >> >> >> > + unclean system shutdown.
> >> >> >> > ++
> >> >> >> > +* `none` disables fsync completely. This must be specified alone.
> >> >> >> > +* `loose-object` hardens objects added to the repo in loose-object form.
> >> >> >> > +* `pack` hardens objects added to the repo in packfile form.
> >> >> >> > +* `pack-metadata` hardens packfile bitmaps and indexes.
> >> >> >> > +* `commit-graph` hardens the commit graph file.
> >> >> >> > +* `objects` is an aggregate option that includes `loose-objects`, `pack`,
> >> >> >> > + `pack-metadata`, and `commit-graph`.
> >> >> >> > +* `default` is an aggregate option that is equivalent to `objects,-loose-object`
> >> >> >> > +* `all` is an aggregate option that syncs all individual components above.
> >> >> >> > +
> >> >> >>
> >> >> >> It's probably a *bit* more work to set up, but I wonder if this wouldn't
> >> >> >> be simpler if we just said (and this is partially going against what I
> >> >> >> noted above):
> >> >> >>
> >> >> >> == BEGIN DOC
> >> >> >>
> >> >> >> core.fsync is a multi-value config variable where each item is a
> >> >> >> pathspec that'll get matched the same way as 'git-ls-files' et al.
> >> >> >>
> >> >> >> When we sync pretend that a path like .git/objects/de/adbeef... is
> >> >> >> relative to the top-level of the git
> >> >> >> directory. E.g. "objects/de/adbeaf.." or "objects/pack/...".
> >> >> >>
> >> >> >> You can then supply a list of wildcards and exclusions to configure
> >> >> >> syncing. or "false", "off" etc. to turn it off. These are synonymous
> >> >> >> with:
> >> >> >>
> >> >> >> ; same as "false"
> >> >> >> core.fsync = ":!*"
> >> >> >>
> >> >> >> Or:
> >> >> >>
> >> >> >> ; same as "true"
> >> >> >> core.fsync = "*"
> >> >> >>
> >> >> >> Or, to selectively sync some things and not others:
> >> >> >>
> >> >> >> ;; Sync objects, but not "info"
> >> >> >> core.fsync = ":!objects/info/**"
> >> >> >> core.fsync = "objects/**"
> >> >> >>
> >> >> >> See gitrepository-layout(5) for details about what sort of paths you
> >> >> >> might be expected to match. Not all paths listed there will go through
> >> >> >> this mechanism (e.g. currently objects do, but nothing to do with config
> >> >> >> does).
> >> >> >>
> >> >> >> We can and will match this against "fake paths", e.g. when writing out
> >> >> >> packs we may match against just the string "objects/pack", we're not
> >> >> >> going to re-check if every packfile we're writing matches your globs,
> >> >> >> ditto for loose objects. Be reasonable!
> >> >> >>
> >> >> >> This metharism is intended as a shorthand that provides some flexibility
> >> >> >> when fsyncing, while not forcing git to come up with labels for all
> >> >> >> paths the git dir, or to support crazyness like "objects/de/adbeef*"
> >> >> >>
> >> >> >> More paths may be added or removed in the future, and we make no
> >> >> >> promises that we won't move things around, so if in doubt use
> >> >> >> e.g. "true" or a wide pattern match like "objects/**". When in doubt
> >> >> >> stick to the golden path of examples provided in this documentation.
> >> >> >>
> >> >> >> == END DOC
> >> >> >>
> >> >> >>
> >> >> >> It's a tad more complex to set up, but I wonder if that isn't worth
> >> >> >> it. It nicely gets around any current and future issues of deciding what
> >> >> >> labels such as "loose-object" etc. to pick, as well as slotting into an
> >> >> >> existing method of doing exclude/include lists.
> >> >> >>
> >> >> >
> >> >> > I think this proposal is a lot of complexity to avoid coming up with a
> >> >> > new name for syncable things as they are added to Git. A path based
> >> >> > mechanism makes it hard to document for the (advanced) user what the
> >> >> > full set of things is and how it might change from release to release.
> >> >> > I think the current core.fsync scheme is a bit easier to understand,
> >> >> > query, and extend.
> >> >>
> >> >> We document it in gitrepository-layout(5). Yeah it has some
> >> >> disadvantages, but one advantage is that you could make the
> >> >> composability easy. I.e. if last exclude wins then a setting of:
> >> >>
> >> >> core.fsync = ":!*"
> >> >> core.fsync = "objects/**"
> >> >>
> >> >> Would reset all previous matches & only match objects/**.
> >> >>
> >> >
> >> > The value of changing this is predicated on taking your previous
> >> > multi-valued config proposal, which I'm still not at all convinced
> >> > about.
> >>
> >> They're orthagonal. I.e. you get benefits from multi-value with or
> >> without this globbing mechanism.
> >>
> >> In any case, I don't feel strongly about/am really advocating this
> >> globbing mechanism. I just wondered if it wouldn't make things simpler
> >> since it would sidestep the need to create any sort of categories for
> >> subsets of gitrepository-layout(5), but maybe not...
> >>
> >> > The schema in the current (v1-v2) version of the patch already
> >> > includes an example of extending the list of syncable things, and
> >> > Patrick Steinhardt made it clear that he feels comfortable adding
> >> > 'refs' to the same schema in a future change.
> >> >
> >> > I'll also emphasize that we're talking about a non-functional,
> >> > relatively corner-case behavioral configuration. These values don't
> >> > change how git's interface behaves except when the system crashes
> >> > during a git command or shortly after one completes.
> >>
> >> That may be something some OS's promise, but it's not something fsync()
> >> or POSIX promises. I.e. you might write a ref, but unless you fsync and
> >> the relevant dir entries another process might not see the update, crash
> >> or not.
> >>
> >
> > I haven't seen any indication that POSIX requires an fsync for
> > visiblity within a running system. I looked at the doc for open,
> > write, and fsync, and saw no indication that it's posix compliant to
> > require an fsync for visibility. I think an OS that required fsync
> > for cross-process visiblity would fail to run Git for a myriad of
> > other reasons and would likely lose all its users. I'm curious where
> > you've seen documentation that allows such unhelpful behavior?
>
> There's multiple unrelated and related things in this area. One is a
> case where you'll e.g. write a file "foo" using stdio, spawn a program
> to work on it in the same program, but it might not see it at all, or
> see empty content, the latter being because you haven't flushed your I/O
> buffers (which you can do via fsync()).
>
For stdio you need to use fflush(3), which just flushes the C
runtime's internal buffers. You need to do the following to do a full
durable write using stdio:
```
FILE *fp;
...
fflush(fp);
fsync(fileno(fp))
```
> The former is that on *nix systems you're generally only guaranteed to
> write to a fd, but not to have the associated metadata be synced for
> you.
>
> That is spelled out e.g. in the DESCRIPTION section of linux's fsync()
> manpage: https://man7.org/linux/man-pages/man2/fdatasync.2.html
>
> I don't know how much you follow non-Windows FS development, but there
> was also a very well known "incident" early in ext4 where it leaned into
> some permissive-by-POSIX behavior that caused data loss in practice on
> buggy programs that didn't correctly use fsync() , since various tooling
> had come to expect the stricter behavior of ext3:
> https://lwn.net/Articles/328363/
>
> That was explicitly in the area of fs metadata being discussed here.
>
> Generally you can expect your VFS layer to be forgiving when it comes to
> IPC, but even that is out the window when it comes to networked
> filesystems, e.g. a shared git repository hosted on NFS.
>
Everything in the fsync(2) DESCRIPTION section is about what data and
metadata reaches the disk (versus just being cached in-memory). I've
become a bit familiar with the ext3 vs ext4 (and delayed alloc)
behavior while researching this feature. These behaviors are all
around the durability you get in the case of kernel crash,
power-failure, or other forms of dirty dismount.
NFS is a complex story. I'm not intimately familiar with its
particular pitfalls, but from looking at the Linux NFS faq
(http://nfs.sourceforge.net/), it appears that a given single NFS
client will remain coherent with itself. Multiple NFS clients
accessing a single Git repo concurrently are probably going to see
some inconsistency. In that kind of case, fsync would help, perhaps,
since it would force NFS clients to issue a COMMIT command to the
server.
> >> That's an aside from these config design questions, and I think
> >> most/(all?) OS's anyone cares about these days tend to make that
> >> implicit promise as part of their VFS behavior, but we should probably
> >> design such an interface to fsync() with such pedantic portability in
> >> mind.
> >
> > Why? To be rude to such a hypothetical system, if a system were so
> > insanely designed, it would be nuts to support it.
>
> Because we know that right now the system calls we're invoking aren't
> guaranteed to store data persistently to disk portably, although they do
> so in practice on most modern OS's.
>
> We're portably to a lot of platforms, and also need to keep e.g. NFS in
> mind, so being able to ask for a pedantic mode when you care about data
> retention at the cost of performance would be nice.
>
> And because the fsync config mode you're proposing is thoroughly
> non-standard, but is known to me much faster by leaning into known
> attributes of specific FS's on specific OS's, if we're not running on
> those it would be sensible to fall back to a stricter mode of
> operation. E.g. syncing all 100 loose objects we just wrote, not just
> the last one.
>
> >> > While you may not personally love the proposed configuration
> >> > interface, I'd want your view on some questions:
> >> > 1. Is it easy for the (advanced) user to set a configuration?
> >> > 2. Is it easy for the (advanced) user to see what was configured?
> >> > 3. Is it easy for the Git community to build on this as we want to add
> >> > things to the list of things to sync?
> >> > a) Is there a good best practice configuration so that people can
> >> > avoid losing integrity for new stuff that they are intending to sync.
> >> > b) If someone has a custom configuration, can that custom
> >> > configuration do something reasonable as they upgrade versions of Git?
> >> > ** In response to this question, I might see some value
> >> > in adding a 'derived-metadata' aggregate that can be disabled so that
> >> > a custom configuration can exclude those as they change version to
> >> > version.
> >> > c) Is it too much maintenance overhead to consider how to present
> >> > this configuration knob for any new hashfile or other datafile in the
> >> > git repo?
> >> > 4. Is there a good path forward to change the default syncable set,
> >> > both in git-for-windows and in Git for other platforms?
> >>
> >> I'm not really sure this globbing this is a good idea, as noted above
> >> just a suggestion etc.
> >>
> >> As noted there it just gets you out of the business of re-defining
> >> gitrepository-layout(5), and assuming too much in advance about certain
> >> use-cases.
> >>
> >> E.g. even "refs" might be too broad for some. I don't tend to be I/O
> >> limited, but I could see how someone who would be would care about
> >> refs/heads but not refs/remotes, or want to exclude logs/* but not the
> >> refs updates themselves etc.
> >
> > This use-case is interesting (distinguishing remote refs from local
> > refs). I think the difficulty of verifying (for even an advanced
> > user) that the right fsyncing is actually happening still puts me on
> > the side of having a carefully curated and documented set of syncable
> > things rather than a file-path-based mechanism.
> >
> > Is this meaningful in the presumably nearby future world of the refsdb
> > backend? Is that somehow split by remote versus local?
>
> There is the upcoming "reftable" work, but that's probably 2-3 years out
> at the earliest for series production workloads in git.git.
>
> >> >> >> > diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
> >> >> >> > index 857be7826f3..916c55d6ce9 100644
> >> >> >> > --- a/builtin/pack-objects.c
> >> >> >> > +++ b/builtin/pack-objects.c
> >> >> >> > @@ -1204,11 +1204,13 @@ static void write_pack_file(void)
> >> >> >> > * If so, rewrite it like in fast-import
> >> >> >> > */
> >> >> >> > if (pack_to_stdout) {
> >> >> >> > - finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_CLOSE);
> >> >> >> > + finalize_hashfile(f, hash, FSYNC_COMPONENT_NONE,
> >> >> >> > + CSUM_HASH_IN_STREAM | CSUM_CLOSE);
> >> >> >>
> >> >> >> Not really related to this per-se, but since you're touching the API
> >> >> >> everything goes through I wonder if callers should just always try to
> >> >> >> fsync, and we can just catch EROFS and EINVAL in the wrapper if someone
> >> >> >> tries to flush stdout, or catch the fd at that lower level.
> >> >> >>
> >> >> >> Or maybe there's a good reason for this...
> >> >> >
> >> >> > It's platform dependent, but I'd expect fsync would do something for
> >> >> > pipes or stdout redirected to a file. In these cases we really don't
> >> >> > want to fsync since we have no idea what we're talking to and we're
> >> >> > potentially worsening performance for probably no benefit.
> >> >>
> >> >> Yeah maybe we should just leave it be.
> >> >>
> >> >> I'd think the C library returning EINVAL would be a trivial performance
> >> >> cost though.
> >> >>
> >> >> It just seemed odd to hardcode assumptions about what can and can't be
> >> >> synced when the POSIX defined function will also tell us that.
> >> >>
> >> >
> >> > Redirecting stdout to a file seems like a common usage for this
> >> > command. That would definitely be fsyncable, but Git has no idea what
> >> > its proper category is since there's no way to know the purpose or
> >> > lifetime of the packfile. I'm going to leave this be, because I'd
> >> > posit that "can it be fsynced?" is not the same as "should it be
> >> > fsynced?". The latter question can't be answered for stdout.
> >>
> >> As noted this was just an aside, and I don't even know if any OS would
> >> do anything meaningful with an fsync() of such a FD anyway.
> >>
> >
> > The underlying fsync primitive does have a meaning on Windows for
> > pipes, but it's certainly not what Git would want to do. Also if
> > stdout is redirected to a file, I'm pretty sure that UNIX OSes would
> > respect the fsync call. However it's not meaningful in the sense of
> > the git repository, since we don't know what the packfile is or why it
> > was created.
>
> I suggested that because I think it's probably nonsensical, but it's
> nonsense that POSIX seems to explicitly tell us that it'll handle
> (probably by silently doing nothing). So in terms of our interface we
> could lean into that and avoid our own special-casing.
>
> >> I just don't see why we wouldn't say:
> >>
> >> 1. We're syncing this category of thing
> >> 2. Try it
> >> 3. If fsync returns "can't fsync that sort of thing" move on
> >>
> >> As opposed to trying to shortcut #3 by doing the detection ourselves.
> >>
> >> I.e. maybe there was a good reason, but it seemed to be some easy
> >> potential win for more simplification, since you were re-doing and
> >> simplifying some of the interface anyway...
> >
> > We're trying to be deliberate about what we're fsyncing. Fsyncing an
> > unknown file created by the packfile code doesn't move us in that
> > direction. In your taxonomy we don't know (1), "what is this category
> > of thing?" Sure it's got the packfile format, but is not known to be
> > an actual packfile that's part of the repository.
>
> We know it's a fd, isn't that sufficient? In any case, I'm fine with
> also keeping it as is, I don't mean to split hairs here.
>
> It just stuck out as an odd part of the interface, why treat some fd's
> specially, instead of just throwing it all at the OS. Presumably the
> first thing the OS will do is figure out if it's a syncable fd or not,
> and act appropriately.
>
I'll put the following comment in pack-objects.c:
/*
* We never fsync when writing to stdout since we may
* not be writing to a specific file. For instance, the
* upload-pack code passes a pipe here. Calling fsync
* on a pipe results in unnecessary synchronization with
* the reader on some platforms.
*/
> >> >>
> >> >> >> > [...]
> >> >> >> > +/*
> >> >> >> > + * These values are used to help identify parts of a repository to fsync.
> >> >> >> > + * FSYNC_COMPONENT_NONE identifies data that will not be a persistent part of the
> >> >> >> > + * repository and so shouldn't be fsynced.
> >> >> >> > + */
> >> >> >> > +enum fsync_component {
> >> >> >> > + FSYNC_COMPONENT_NONE = 0,
> >> >> >>
> >> >> >> I haven't read ahead much but in most other such cases we don't define
> >> >> >> the "= 0", just start at 1<<0, then check the flags elsewhere...
> >> >> >>
> >> >> >> > +static const struct fsync_component_entry {
> >> >> >> > + const char *name;
> >> >> >> > + enum fsync_component component_bits;
> >> >> >> > +} fsync_component_table[] = {
> >> >> >> > + { "loose-object", FSYNC_COMPONENT_LOOSE_OBJECT },
> >> >> >> > + { "pack", FSYNC_COMPONENT_PACK },
> >> >> >> > + { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA },
> >> >> >> > + { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH },
> >> >> >> > + { "objects", FSYNC_COMPONENTS_OBJECTS },
> >> >> >> > + { "default", FSYNC_COMPONENTS_DEFAULT },
> >> >> >> > + { "all", FSYNC_COMPONENTS_ALL },
> >> >> >> > +};
> >> >> >> > +
> >> >> >> > +static enum fsync_component parse_fsync_components(const char *var, const char *string)
> >> >> >> > +{
> >> >> >> > + enum fsync_component output = 0;
> >> >> >> > +
> >> >> >> > + if (!strcmp(string, "none"))
> >> >> >> > + return output;
> >> >> >> > +
> >> >> >> > + while (string) {
> >> >> >> > + int i;
> >> >> >> > + size_t len;
> >> >> >> > + const char *ep;
> >> >> >> > + int negated = 0;
> >> >> >> > + int found = 0;
> >> >> >> > +
> >> >> >> > + string = string + strspn(string, ", \t\n\r");
> >> >> >>
> >> >> >> Aside from the "use a list" isn't this hardcoding some windows-specific
> >> >> >> assumptions with \n\r? Maybe not...
> >> >> >
> >> >> > I shamelessly stole this code from parse_whitespace_rule. I thought
> >> >> > about making a helper to be called by both functions, but the amount
> >> >> > of state going into and out of the wrapper via arguments was
> >> >> > substantial and seemed to negate the benefit of deduplication.
> >> >>
> >> >> FWIW string_list_split() is easier to work with in those cases, or at
> >> >> least I think so...
> >> >
> >> > This code runs at startup for a variable that may be present on some
> >> > installations. The nice property of the current patch's code is that
> >> > it's already a well-tested pattern that doesn't do any allocations as
> >> > it's working, unlike string_list_split().
> >>
> >> Multi-value config would also get you fewer allocations :)
> >>
> >> Anyway, I mainly meant to point out that for stuff like this it's fine
> >> to optimize it for ease rather than micro-optimize allocations. Those
> >> really aren't a bottleneck on this scale.
> >>
> >> Even in that case there's string_list_split_in_place(), which can be a
> >> bit nicer than manual C-string fiddling.
> >>
> >
> > Am I allowed to change the config value string in place? The
> > core.whitespace code is careful not to modify the string. I kind of
> > like the parse_ws_error_highlight code a little better now that I've
> > seen it, but I think the current code is fine too.
>
> I don't remember offhand if that's safe, probably not. So you'll need a
> copy here.
>
> >> > I hope you know that I appreciate your review feedback, even though
> >> > I'm pushing back on most of it so far this round. I'll be sending v3
> >> > to the list soon after giving it another look over.
> >>
> >> Sure, no worries. Just hoping to help. If you go for something different
> >> etc. that's fine. Just hoping to bridge the gap in some knowledge /
> >> offer potentially interesting suggestions (some of which may be dead
> >> ends, like the config glob thing...).
Thanks again for the review, I'll send an updated PR soon.
Thanks,
Neeraj
^ permalink raw reply [flat|nested] 122+ messages in thread
* [PATCH v4 0/4] A design for future-proofing fsync() configuration
2021-12-09 0:57 ` [PATCH v3 0/4] " Neeraj K. Singh via GitGitGadget
` (4 preceding siblings ...)
2022-01-08 1:13 ` [PATCH v3 0/4] A design for future-proofing fsync() configuration Neeraj Singh
@ 2022-02-01 3:33 ` Neeraj K. Singh via GitGitGadget
2022-02-01 3:33 ` [PATCH v4 1/4] core.fsyncmethod: add writeout-only mode Neeraj Singh via GitGitGadget
` (4 more replies)
5 siblings, 5 replies; 122+ messages in thread
From: Neeraj K. Singh via GitGitGadget @ 2022-02-01 3:33 UTC (permalink / raw)
To: git; +Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps,
Neeraj K. Singh
This is an implementation of an extensible configuration mechanism for
fsyncing persistent components of a repo.
The main goals are to separate the "what" to sync from the "how". There are
now two settings: core.fsync - Control the 'what', including the index.
core.fsyncMethod - Control the 'how'. Currently we support writeout-only and
full fsync.
Syncing of refs can be layered on top of core.fsync. And batch mode will be
layered on core.fsyncMethod.
core.fsyncObjectfiles is removed and will issue a deprecation warning if
it's seen.
I'd like to get agreement on this direction before submitting batch mode to
the list. The batch mode series is available to view at
https://github.com/gitgitgadget/git/pull/1134
Please see [1], [2], and [3] for discussions that led to this series.
After this change, new persistent data files added to the repo will need to
be added to the fsync_component enum and documented in the
Documentation/config/core.txt text.
V4 changes:
* Rebase onto master at b23dac905bd.
* Add a comment to write_pack_file indicating why we don't fsync when
writing to stdout.
* I kept the configuration schema as-is rather than switching to
multi-value. The thinking here is that a stateless last-one-wins config
schema (comma separated) will make it easier to achieve some holistic
self-consistent fsync configuration for a particular repo.
V3 changes:
* Remove relative path from git-compat-util.h include [4].
* Updated newly added warning texts to have more context for localization
[4].
* Fixed tab spacing in enum fsync_action
* Moved the fsync looping out to a helper and do it consistently. [4]
* Changed commit description to use camelCase for config names. [5]
* Add an optional fourth patch with derived-metadata so that the user can
exclude a forward-compatible set of things that should be recomputable
given existing data.
V2 changes:
* Updated the documentation for core.fsyncmethod to be less certain.
writeout-only probably does not do the right thing on Linux.
* Split out the core.fsync=index change into its own commit.
* Rename REPO_COMPONENT to FSYNC_COMPONENT. This is really specific to
fsyncing, so the name should reflect that.
* Re-add missing Makefile change for SYNC_FILE_RANGE.
* Tested writeout-only mode, index syncing, and general config settings.
[1] https://lore.kernel.org/git/211110.86r1bogg27.gmgdl@evledraar.gmail.com/
[2]
https://lore.kernel.org/git/dd65718814011eb93ccc4428f9882e0f025224a6.1636029491.git.ps@pks.im/
[3]
https://lore.kernel.org/git/pull.1076.git.git.1629856292.gitgitgadget@gmail.com/
[4]
https://lore.kernel.org/git/CANQDOdf8C4-haK9=Q_J4Cid8bQALnmGDm=SvatRbaVf+tkzqLw@mail.gmail.com/
[5] https://lore.kernel.org/git/211207.861r2opplg.gmgdl@evledraar.gmail.com/
Neeraj Singh (4):
core.fsyncmethod: add writeout-only mode
core.fsync: introduce granular fsync control
core.fsync: new option to harden the index
core.fsync: add a `derived-metadata` aggregate option
Documentation/config/core.txt | 35 ++++++++---
Makefile | 6 ++
builtin/fast-import.c | 2 +-
builtin/index-pack.c | 4 +-
builtin/pack-objects.c | 24 +++++---
bulk-checkin.c | 5 +-
cache.h | 49 +++++++++++++++-
commit-graph.c | 3 +-
compat/mingw.h | 3 +
compat/win32/flush.c | 28 +++++++++
config.c | 90 ++++++++++++++++++++++++++++-
config.mak.uname | 3 +
configure.ac | 8 +++
contrib/buildsystems/CMakeLists.txt | 3 +-
csum-file.c | 5 +-
csum-file.h | 3 +-
environment.c | 3 +-
git-compat-util.h | 24 ++++++++
midx.c | 3 +-
object-file.c | 3 +-
pack-bitmap-write.c | 3 +-
pack-write.c | 13 +++--
read-cache.c | 19 ++++--
wrapper.c | 64 ++++++++++++++++++++
write-or-die.c | 11 ++--
25 files changed, 367 insertions(+), 47 deletions(-)
create mode 100644 compat/win32/flush.c
base-commit: b23dac905bde28da47543484320db16312c87551
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1093%2Fneerajsi-msft%2Fns%2Fcore-fsync-v4
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1093/neerajsi-msft/ns/core-fsync-v4
Pull-Request: https://github.com/gitgitgadget/git/pull/1093
Range-diff vs v3:
1: 15edfe51509 ! 1: 51a218d100d core.fsyncmethod: add writeout-only mode
@@ Makefile: ifdef HAVE_CLOCK_MONOTONIC
endif
## cache.h ##
-@@ cache.h: extern int read_replace_refs;
- extern char *git_replace_ref_base;
+@@ cache.h: extern char *git_replace_ref_base;
extern int fsync_object_files;
+ extern int use_fsync;
+
+enum fsync_method {
+ FSYNC_METHOD_FSYNC,
@@ compat/win32/flush.c (new)
+
+#define FLUSH_FLAGS_FILE_DATA_ONLY 1
+
-+ DECLARE_PROC_ADDR(ntdll.dll, NTSTATUS, NtFlushBuffersFileEx,
++ DECLARE_PROC_ADDR(ntdll.dll, NTSTATUS, NTAPI, NtFlushBuffersFileEx,
+ HANDLE FileHandle, ULONG Flags, PVOID Parameters, ULONG ParameterSize,
+ PIO_STATUS_BLOCK IoStatusBlock);
+
@@ contrib/buildsystems/CMakeLists.txt: if(CMAKE_SYSTEM_NAME STREQUAL "Windows")
compat/nedmalloc/nedmalloc.c compat/strdup.c)
## environment.c ##
-@@ environment.c: const char *git_attributes_file;
- const char *git_hooks_path;
- int zlib_compression_level = Z_BEST_SPEED;
+@@ environment.c: int zlib_compression_level = Z_BEST_SPEED;
int pack_compression_level = Z_DEFAULT_COMPRESSION;
--int fsync_object_files;
+ int fsync_object_files;
+ int use_fsync = -1;
+enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT;
size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE;
size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT;
@@ wrapper.c: int xmkstemp_mode(char *filename_template, int mode)
int err;
## write-or-die.c ##
-@@ write-or-die.c: void fprintf_or_die(FILE *f, const char *fmt, ...)
-
- void fsync_or_die(int fd, const char *msg)
- {
+@@ write-or-die.c: void fsync_or_die(int fd, const char *msg)
+ use_fsync = git_env_bool("GIT_TEST_FSYNC", 1);
+ if (!use_fsync)
+ return;
- while (fsync(fd) < 0) {
- if (errno != EINTR)
- die_errno("fsync error on '%s'", msg);
- }
++
+ if (fsync_method == FSYNC_METHOD_WRITEOUT_ONLY &&
+ git_fsync(fd, FSYNC_WRITEOUT_ONLY) >= 0)
+ return;
2: 080be1a6f64 ! 2: 7a164ba9571 core.fsync: introduce granular fsync control
@@ builtin/index-pack.c: static void final(const char *final_pack_name, const char
## builtin/pack-objects.c ##
@@ builtin/pack-objects.c: static void write_pack_file(void)
- * If so, rewrite it like in fast-import
- */
+ display_progress(progress_state, written);
+ }
+
+- /*
+- * Did we write the wrong # entries in the header?
+- * If so, rewrite it like in fast-import
+- */
if (pack_to_stdout) {
- finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_CLOSE);
++ /*
++ * We never fsync when writing to stdout since we may
++ * not be writing to an actual pack file. For instance,
++ * the upload-pack code passes a pipe here. Calling
++ * fsync on a pipe results in unnecessary
++ * synchronization with the reader on some platforms.
++ */
+ finalize_hashfile(f, hash, FSYNC_COMPONENT_NONE,
+ CSUM_HASH_IN_STREAM | CSUM_CLOSE);
} else if (nr_written == nr_remaining) {
@@ builtin/pack-objects.c: static void write_pack_file(void)
+ CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
} else {
- int fd = finalize_hashfile(f, hash, 0);
++ /*
++ * If we wrote the wrong number of entries in the
++ * header, rewrite it like in fast-import.
++ */
++
+ int fd = finalize_hashfile(f, hash, FSYNC_COMPONENT_PACK, 0);
fixup_pack_header_footer(fd, hash, pack_tmp_name,
nr_written, hash, offset);
@@ cache.h: void reset_shared_repository(void);
extern char *git_replace_ref_base;
-extern int fsync_object_files;
+-extern int use_fsync;
+/*
+ * These values are used to help identify parts of a repository to fsync.
+ * FSYNC_COMPONENT_NONE identifies data that will not be a persistent part of the
+ * repository and so shouldn't be fsynced.
+ */
+enum fsync_component {
-+ FSYNC_COMPONENT_NONE = 0,
++ FSYNC_COMPONENT_NONE,
+ FSYNC_COMPONENT_LOOSE_OBJECT = 1 << 0,
+ FSYNC_COMPONENT_PACK = 1 << 1,
+ FSYNC_COMPONENT_PACK_METADATA = 1 << 2,
@@ cache.h: void reset_shared_repository(void);
enum fsync_method {
FSYNC_METHOD_FSYNC,
+@@ cache.h: enum fsync_method {
+ };
+
+ extern enum fsync_method fsync_method;
++extern int use_fsync;
+ extern int core_preload_index;
+ extern int precomposed_unicode;
+ extern int protect_hfs;
@@ cache.h: int copy_file_with_time(const char *dst, const char *src, int mode);
void write_or_die(int fd, const void *buf, size_t count);
void fsync_or_die(int fd, const char *);
@@ csum-file.h: int hashfile_truncate(struct hashfile *, struct hashfile_checkpoint
void crc32_begin(struct hashfile *);
## environment.c ##
-@@ environment.c: const char *git_hooks_path;
+@@ environment.c: const char *git_attributes_file;
+ const char *git_hooks_path;
int zlib_compression_level = Z_BEST_SPEED;
int pack_compression_level = Z_DEFAULT_COMPRESSION;
+-int fsync_object_files;
+ int use_fsync = -1;
enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT;
+enum fsync_component fsync_components = FSYNC_COMPONENTS_DEFAULT;
size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE;
@@ midx.c: static int write_midx_internal(const char *object_dir,
## object-file.c ##
@@ object-file.c: int hash_object_file(const struct git_hash_algo *algo, const void *buf,
- /* Finalize a file on disk, and close it. */
static void close_loose_object(int fd)
{
-- if (fsync_object_files)
-- fsync_or_die(fd, "loose object file");
-+ fsync_component_or_die(FSYNC_COMPONENT_LOOSE_OBJECT, fd, "loose object file");
+ if (!the_repository->objects->odb->will_destroy) {
+- if (fsync_object_files)
+- fsync_or_die(fd, "loose object file");
++ fsync_component_or_die(FSYNC_COMPONENT_LOOSE_OBJECT, fd, "loose object file");
+ }
+
if (close(fd) != 0)
- die_errno(_("error when closing loose object file"));
- }
## pack-bitmap-write.c ##
@@ pack-bitmap-write.c: void bitmap_writer_finish(struct pack_idx_entry **index,
3: 2207950beba = 3: f217dba77a1 core.fsync: new option to harden the index
4: a830d177d4c = 4: 5c22a41c1f3 core.fsync: add a `derived-metadata` aggregate option
--
gitgitgadget
^ permalink raw reply [flat|nested] 122+ messages in thread
* [PATCH v4 1/4] core.fsyncmethod: add writeout-only mode
2022-02-01 3:33 ` [PATCH v4 " Neeraj K. Singh via GitGitGadget
@ 2022-02-01 3:33 ` Neeraj Singh via GitGitGadget
2022-02-01 3:33 ` [PATCH v4 2/4] core.fsync: introduce granular fsync control Neeraj Singh via GitGitGadget
` (3 subsequent siblings)
4 siblings, 0 replies; 122+ messages in thread
From: Neeraj Singh via GitGitGadget @ 2022-02-01 3:33 UTC (permalink / raw)
To: git
Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps,
Neeraj K. Singh, Neeraj Singh
From: Neeraj Singh <neerajsi@microsoft.com>
This commit introduces the `core.fsyncMethod` configuration
knob, which can currently be set to `fsync` or `writeout-only`.
The new writeout-only mode attempts to tell the operating system to
flush its in-memory page cache to the storage hardware without issuing a
CACHE_FLUSH command to the storage controller.
Writeout-only fsync is significantly faster than a vanilla fsync on
common hardware, since data is written to a disk-side cache rather than
all the way to a durable medium. Later changes in this patch series will
take advantage of this primitive to implement batching of hardware
flushes.
When git_fsync is called with FSYNC_WRITEOUT_ONLY, it may fail and the
caller is expected to do an ordinary fsync as needed.
On Apple platforms, the fsync system call does not issue a CACHE_FLUSH
directive to the storage controller. This change updates fsync to do
fcntl(F_FULLFSYNC) to make fsync actually durable. We maintain parity
with existing behavior on Apple platforms by setting the default value
of the new core.fsyncMethod option.
Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
---
Documentation/config/core.txt | 9 ++++
Makefile | 6 +++
cache.h | 7 ++++
compat/mingw.h | 3 ++
compat/win32/flush.c | 28 +++++++++++++
config.c | 12 ++++++
config.mak.uname | 3 ++
configure.ac | 8 ++++
contrib/buildsystems/CMakeLists.txt | 3 +-
environment.c | 1 +
git-compat-util.h | 24 +++++++++++
wrapper.c | 64 +++++++++++++++++++++++++++++
write-or-die.c | 11 +++--
13 files changed, 174 insertions(+), 5 deletions(-)
create mode 100644 compat/win32/flush.c
diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
index c04f62a54a1..dbb134f7136 100644
--- a/Documentation/config/core.txt
+++ b/Documentation/config/core.txt
@@ -547,6 +547,15 @@ core.whitespace::
is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent`
errors. The default tab width is 8. Allowed values are 1 to 63.
+core.fsyncMethod::
+ A value indicating the strategy Git will use to harden repository data
+ using fsync and related primitives.
++
+* `fsync` uses the fsync() system call or platform equivalents.
+* `writeout-only` issues pagecache writeback requests, but depending on the
+ filesystem and storage hardware, data added to the repository may not be
+ durable in the event of a system crash. This is the default mode on macOS.
+
core.fsyncObjectFiles::
This boolean will enable 'fsync()' when writing object files.
+
diff --git a/Makefile b/Makefile
index 5580859afdb..1eff9953280 100644
--- a/Makefile
+++ b/Makefile
@@ -405,6 +405,8 @@ all::
#
# Define HAVE_CLOCK_MONOTONIC if your platform has CLOCK_MONOTONIC.
#
+# Define HAVE_SYNC_FILE_RANGE if your platform has sync_file_range.
+#
# Define NEEDS_LIBRT if your platform requires linking with librt (glibc version
# before 2.17) for clock_gettime and CLOCK_MONOTONIC.
#
@@ -1892,6 +1894,10 @@ ifdef HAVE_CLOCK_MONOTONIC
BASIC_CFLAGS += -DHAVE_CLOCK_MONOTONIC
endif
+ifdef HAVE_SYNC_FILE_RANGE
+ BASIC_CFLAGS += -DHAVE_SYNC_FILE_RANGE
+endif
+
ifdef NEEDS_LIBRT
EXTLIBS += -lrt
endif
diff --git a/cache.h b/cache.h
index 281f00ab1b1..37a32034b2f 100644
--- a/cache.h
+++ b/cache.h
@@ -995,6 +995,13 @@ extern char *git_replace_ref_base;
extern int fsync_object_files;
extern int use_fsync;
+
+enum fsync_method {
+ FSYNC_METHOD_FSYNC,
+ FSYNC_METHOD_WRITEOUT_ONLY
+};
+
+extern enum fsync_method fsync_method;
extern int core_preload_index;
extern int precomposed_unicode;
extern int protect_hfs;
diff --git a/compat/mingw.h b/compat/mingw.h
index c9a52ad64a6..6074a3d3ced 100644
--- a/compat/mingw.h
+++ b/compat/mingw.h
@@ -329,6 +329,9 @@ int mingw_getpagesize(void);
#define getpagesize mingw_getpagesize
#endif
+int win32_fsync_no_flush(int fd);
+#define fsync_no_flush win32_fsync_no_flush
+
struct rlimit {
unsigned int rlim_cur;
};
diff --git a/compat/win32/flush.c b/compat/win32/flush.c
new file mode 100644
index 00000000000..291f90ea940
--- /dev/null
+++ b/compat/win32/flush.c
@@ -0,0 +1,28 @@
+#include "git-compat-util.h"
+#include <winternl.h>
+#include "lazyload.h"
+
+int win32_fsync_no_flush(int fd)
+{
+ IO_STATUS_BLOCK io_status;
+
+#define FLUSH_FLAGS_FILE_DATA_ONLY 1
+
+ DECLARE_PROC_ADDR(ntdll.dll, NTSTATUS, NTAPI, NtFlushBuffersFileEx,
+ HANDLE FileHandle, ULONG Flags, PVOID Parameters, ULONG ParameterSize,
+ PIO_STATUS_BLOCK IoStatusBlock);
+
+ if (!INIT_PROC_ADDR(NtFlushBuffersFileEx)) {
+ errno = ENOSYS;
+ return -1;
+ }
+
+ memset(&io_status, 0, sizeof(io_status));
+ if (NtFlushBuffersFileEx((HANDLE)_get_osfhandle(fd), FLUSH_FLAGS_FILE_DATA_ONLY,
+ NULL, 0, &io_status)) {
+ errno = EINVAL;
+ return -1;
+ }
+
+ return 0;
+}
diff --git a/config.c b/config.c
index 2bffa8d4a01..f67f545f839 100644
--- a/config.c
+++ b/config.c
@@ -1490,6 +1490,18 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
return 0;
}
+ if (!strcmp(var, "core.fsyncmethod")) {
+ if (!value)
+ return config_error_nonbool(var);
+ if (!strcmp(value, "fsync"))
+ fsync_method = FSYNC_METHOD_FSYNC;
+ else if (!strcmp(value, "writeout-only"))
+ fsync_method = FSYNC_METHOD_WRITEOUT_ONLY;
+ else
+ warning(_("ignoring unknown core.fsyncMethod value '%s'"), value);
+
+ }
+
if (!strcmp(var, "core.fsyncobjectfiles")) {
fsync_object_files = git_config_bool(var, value);
return 0;
diff --git a/config.mak.uname b/config.mak.uname
index c48db45106c..2c67b3b93ce 100644
--- a/config.mak.uname
+++ b/config.mak.uname
@@ -57,6 +57,7 @@ ifeq ($(uname_S),Linux)
HAVE_CLOCK_MONOTONIC = YesPlease
# -lrt is needed for clock_gettime on glibc <= 2.16
NEEDS_LIBRT = YesPlease
+ HAVE_SYNC_FILE_RANGE = YesPlease
HAVE_GETDELIM = YesPlease
FREAD_READS_DIRECTORIES = UnfortunatelyYes
BASIC_CFLAGS += -DHAVE_SYSINFO
@@ -462,6 +463,7 @@ endif
CFLAGS =
BASIC_CFLAGS = -nologo -I. -Icompat/vcbuild/include -DWIN32 -D_CONSOLE -DHAVE_STRING_H -D_CRT_SECURE_NO_WARNINGS -D_CRT_NONSTDC_NO_DEPRECATE
COMPAT_OBJS = compat/msvc.o compat/winansi.o \
+ compat/win32/flush.o \
compat/win32/path-utils.o \
compat/win32/pthread.o compat/win32/syslog.o \
compat/win32/trace2_win32_process_info.o \
@@ -639,6 +641,7 @@ ifeq ($(uname_S),MINGW)
COMPAT_CFLAGS += -DSTRIP_EXTENSION=\".exe\"
COMPAT_OBJS += compat/mingw.o compat/winansi.o \
compat/win32/trace2_win32_process_info.o \
+ compat/win32/flush.o \
compat/win32/path-utils.o \
compat/win32/pthread.o compat/win32/syslog.o \
compat/win32/dirent.o
diff --git a/configure.ac b/configure.ac
index d60d494ee4c..660b91f90b4 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1095,6 +1095,14 @@ AC_COMPILE_IFELSE([CLOCK_MONOTONIC_SRC],
[AC_MSG_RESULT([no])
HAVE_CLOCK_MONOTONIC=])
GIT_CONF_SUBST([HAVE_CLOCK_MONOTONIC])
+
+#
+# Define HAVE_SYNC_FILE_RANGE=YesPlease if sync_file_range is available.
+GIT_CHECK_FUNC(sync_file_range,
+ [HAVE_SYNC_FILE_RANGE=YesPlease],
+ [HAVE_SYNC_FILE_RANGE])
+GIT_CONF_SUBST([HAVE_SYNC_FILE_RANGE])
+
#
# Define NO_SETITIMER if you don't have setitimer.
GIT_CHECK_FUNC(setitimer,
diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt
index 5100f56bb37..276e74c1d54 100644
--- a/contrib/buildsystems/CMakeLists.txt
+++ b/contrib/buildsystems/CMakeLists.txt
@@ -261,7 +261,8 @@ if(CMAKE_SYSTEM_NAME STREQUAL "Windows")
NOGDI OBJECT_CREATION_MODE=1 __USE_MINGW_ANSI_STDIO=0
USE_NED_ALLOCATOR OVERRIDE_STRDUP MMAP_PREVENTS_DELETE USE_WIN32_MMAP
UNICODE _UNICODE HAVE_WPGMPTR ENSURE_MSYSTEM_IS_SET)
- list(APPEND compat_SOURCES compat/mingw.c compat/winansi.c compat/win32/path-utils.c
+ list(APPEND compat_SOURCES compat/mingw.c compat/winansi.c
+ compat/win32/flush.c compat/win32/path-utils.c
compat/win32/pthread.c compat/win32mmap.c compat/win32/syslog.c
compat/win32/trace2_win32_process_info.c compat/win32/dirent.c
compat/nedmalloc/nedmalloc.c compat/strdup.c)
diff --git a/environment.c b/environment.c
index fd0501e77a5..3e3620d759f 100644
--- a/environment.c
+++ b/environment.c
@@ -44,6 +44,7 @@ int zlib_compression_level = Z_BEST_SPEED;
int pack_compression_level = Z_DEFAULT_COMPRESSION;
int fsync_object_files;
int use_fsync = -1;
+enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT;
size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE;
size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT;
size_t delta_base_cache_limit = 96 * 1024 * 1024;
diff --git a/git-compat-util.h b/git-compat-util.h
index 1229c8296b9..76cb85a56b1 100644
--- a/git-compat-util.h
+++ b/git-compat-util.h
@@ -1265,6 +1265,30 @@ __attribute__((format (printf, 1, 2))) NORETURN
void BUG(const char *fmt, ...);
#endif
+#ifdef __APPLE__
+#define FSYNC_METHOD_DEFAULT FSYNC_METHOD_WRITEOUT_ONLY
+#else
+#define FSYNC_METHOD_DEFAULT FSYNC_METHOD_FSYNC
+#endif
+
+enum fsync_action {
+ FSYNC_WRITEOUT_ONLY,
+ FSYNC_HARDWARE_FLUSH
+};
+
+/*
+ * Issues an fsync against the specified file according to the specified mode.
+ *
+ * FSYNC_WRITEOUT_ONLY attempts to use interfaces available on some operating
+ * systems to flush the OS cache without issuing a flush command to the storage
+ * controller. If those interfaces are unavailable, the function fails with
+ * ENOSYS.
+ *
+ * FSYNC_HARDWARE_FLUSH does an OS writeout and hardware flush to ensure that
+ * changes are durable. It is not expected to fail.
+ */
+int git_fsync(int fd, enum fsync_action action);
+
/*
* Preserves errno, prints a message, but gives no warning for ENOENT.
* Returns 0 on success, which includes trying to unlink an object that does
diff --git a/wrapper.c b/wrapper.c
index 36e12119d76..572f28f14ff 100644
--- a/wrapper.c
+++ b/wrapper.c
@@ -546,6 +546,70 @@ int xmkstemp_mode(char *filename_template, int mode)
return fd;
}
+/*
+ * Some platforms return EINTR from fsync. Since fsync is invoked in some
+ * cases by a wrapper that dies on failure, do not expose EINTR to callers.
+ */
+static int fsync_loop(int fd)
+{
+ int err;
+
+ do {
+ err = fsync(fd);
+ } while (err < 0 && errno == EINTR);
+ return err;
+}
+
+int git_fsync(int fd, enum fsync_action action)
+{
+ switch (action) {
+ case FSYNC_WRITEOUT_ONLY:
+
+#ifdef __APPLE__
+ /*
+ * on macOS, fsync just causes filesystem cache writeback but does not
+ * flush hardware caches.
+ */
+ return fsync_loop(fd);
+#endif
+
+#ifdef HAVE_SYNC_FILE_RANGE
+ /*
+ * On linux 2.6.17 and above, sync_file_range is the way to issue
+ * a writeback without a hardware flush. An offset of 0 and size of 0
+ * indicates writeout of the entire file and the wait flags ensure that all
+ * dirty data is written to the disk (potentially in a disk-side cache)
+ * before we continue.
+ */
+
+ return sync_file_range(fd, 0, 0, SYNC_FILE_RANGE_WAIT_BEFORE |
+ SYNC_FILE_RANGE_WRITE |
+ SYNC_FILE_RANGE_WAIT_AFTER);
+#endif
+
+#ifdef fsync_no_flush
+ return fsync_no_flush(fd);
+#endif
+
+ errno = ENOSYS;
+ return -1;
+
+ case FSYNC_HARDWARE_FLUSH:
+ /*
+ * On some platforms fsync may return EINTR. Try again in this
+ * case, since callers asking for a hardware flush may die if
+ * this function returns an error.
+ */
+#ifdef __APPLE__
+ return fcntl(fd, F_FULLFSYNC);
+#else
+ return fsync_loop(fd);
+#endif
+ default:
+ BUG("unexpected git_fsync(%d) call", action);
+ }
+}
+
static int warn_if_unremovable(const char *op, const char *file, int rc)
{
int err;
diff --git a/write-or-die.c b/write-or-die.c
index a3d5784cec9..9faa5f9f563 100644
--- a/write-or-die.c
+++ b/write-or-die.c
@@ -62,10 +62,13 @@ void fsync_or_die(int fd, const char *msg)
use_fsync = git_env_bool("GIT_TEST_FSYNC", 1);
if (!use_fsync)
return;
- while (fsync(fd) < 0) {
- if (errno != EINTR)
- die_errno("fsync error on '%s'", msg);
- }
+
+ if (fsync_method == FSYNC_METHOD_WRITEOUT_ONLY &&
+ git_fsync(fd, FSYNC_WRITEOUT_ONLY) >= 0)
+ return;
+
+ if (git_fsync(fd, FSYNC_HARDWARE_FLUSH) < 0)
+ die_errno("fsync error on '%s'", msg);
}
void write_or_die(int fd, const void *buf, size_t count)
--
gitgitgadget
^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v4 2/4] core.fsync: introduce granular fsync control
2022-02-01 3:33 ` [PATCH v4 " Neeraj K. Singh via GitGitGadget
2022-02-01 3:33 ` [PATCH v4 1/4] core.fsyncmethod: add writeout-only mode Neeraj Singh via GitGitGadget
@ 2022-02-01 3:33 ` Neeraj Singh via GitGitGadget
2022-02-02 0:51 ` Junio C Hamano
2022-02-01 3:33 ` [PATCH v4 3/4] core.fsync: new option to harden the index Neeraj Singh via GitGitGadget
` (2 subsequent siblings)
4 siblings, 1 reply; 122+ messages in thread
From: Neeraj Singh via GitGitGadget @ 2022-02-01 3:33 UTC (permalink / raw)
To: git
Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps,
Neeraj K. Singh, Neeraj Singh
From: Neeraj Singh <neerajsi@microsoft.com>
This commit introduces the `core.fsync` configuration
knob which can be used to control how components of the
repository are made durable on disk.
This setting allows future extensibility of the list of
syncable components:
* We issue a warning rather than an error for unrecognized
components, so new configs can be used with old Git versions.
* We support negation, so users can choose one of the default
aggregate options and then remove components that they don't
want. The user would then harden any new components added in
a Git version update.
This also supports the common request of doing absolutely no
fysncing with the `core.fsync=none` value, which is expected
to make the test suite faster.
Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
---
Documentation/config/core.txt | 27 +++++++++----
builtin/fast-import.c | 2 +-
builtin/index-pack.c | 4 +-
builtin/pack-objects.c | 24 +++++++----
bulk-checkin.c | 5 ++-
cache.h | 41 ++++++++++++++++++-
commit-graph.c | 3 +-
config.c | 76 ++++++++++++++++++++++++++++++++++-
csum-file.c | 5 ++-
csum-file.h | 3 +-
environment.c | 2 +-
midx.c | 3 +-
object-file.c | 3 +-
pack-bitmap-write.c | 3 +-
pack-write.c | 13 +++---
read-cache.c | 2 +-
16 files changed, 177 insertions(+), 39 deletions(-)
diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
index dbb134f7136..4f1747ec871 100644
--- a/Documentation/config/core.txt
+++ b/Documentation/config/core.txt
@@ -547,6 +547,25 @@ core.whitespace::
is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent`
errors. The default tab width is 8. Allowed values are 1 to 63.
+core.fsync::
+ A comma-separated list of parts of the repository which should be
+ hardened via the core.fsyncMethod when created or modified. You can
+ disable hardening of any component by prefixing it with a '-'. Later
+ items take precedence over earlier ones in the list. For example,
+ `core.fsync=all,-pack-metadata` means "harden everything except pack
+ metadata." Items that are not hardened may be lost in the event of an
+ unclean system shutdown.
++
+* `none` disables fsync completely. This must be specified alone.
+* `loose-object` hardens objects added to the repo in loose-object form.
+* `pack` hardens objects added to the repo in packfile form.
+* `pack-metadata` hardens packfile bitmaps and indexes.
+* `commit-graph` hardens the commit graph file.
+* `objects` is an aggregate option that includes `loose-objects`, `pack`,
+ `pack-metadata`, and `commit-graph`.
+* `default` is an aggregate option that is equivalent to `objects,-loose-object`
+* `all` is an aggregate option that syncs all individual components above.
+
core.fsyncMethod::
A value indicating the strategy Git will use to harden repository data
using fsync and related primitives.
@@ -556,14 +575,6 @@ core.fsyncMethod::
filesystem and storage hardware, data added to the repository may not be
durable in the event of a system crash. This is the default mode on macOS.
-core.fsyncObjectFiles::
- This boolean will enable 'fsync()' when writing object files.
-+
-This is a total waste of time and effort on a filesystem that orders
-data writes properly, but can be useful for filesystems that do not use
-journalling (traditional UNIX filesystems) or that only journal metadata
-and not file contents (OS X's HFS+, or Linux ext3 with "data=writeback").
-
core.preloadIndex::
Enable parallel index preload for operations like 'git diff'
+
diff --git a/builtin/fast-import.c b/builtin/fast-import.c
index 2b2e28bad79..ff70aeb1a0e 100644
--- a/builtin/fast-import.c
+++ b/builtin/fast-import.c
@@ -858,7 +858,7 @@ static void end_packfile(void)
struct tag *t;
close_pack_windows(pack_data);
- finalize_hashfile(pack_file, cur_pack_oid.hash, 0);
+ finalize_hashfile(pack_file, cur_pack_oid.hash, FSYNC_COMPONENT_PACK, 0);
fixup_pack_header_footer(pack_data->pack_fd, pack_data->hash,
pack_data->pack_name, object_count,
cur_pack_oid.hash, pack_size);
diff --git a/builtin/index-pack.c b/builtin/index-pack.c
index 3c2e6aee3cc..b871d721e95 100644
--- a/builtin/index-pack.c
+++ b/builtin/index-pack.c
@@ -1286,7 +1286,7 @@ static void conclude_pack(int fix_thin_pack, const char *curr_pack, unsigned cha
nr_objects - nr_objects_initial);
stop_progress_msg(&progress, msg.buf);
strbuf_release(&msg);
- finalize_hashfile(f, tail_hash, 0);
+ finalize_hashfile(f, tail_hash, FSYNC_COMPONENT_PACK, 0);
hashcpy(read_hash, pack_hash);
fixup_pack_header_footer(output_fd, pack_hash,
curr_pack, nr_objects,
@@ -1508,7 +1508,7 @@ static void final(const char *final_pack_name, const char *curr_pack_name,
if (!from_stdin) {
close(input_fd);
} else {
- fsync_or_die(output_fd, curr_pack_name);
+ fsync_component_or_die(FSYNC_COMPONENT_PACK, output_fd, curr_pack_name);
err = close(output_fd);
if (err)
die_errno(_("error while closing pack file"));
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index ba2006f2212..b483ba65adb 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -1199,16 +1199,26 @@ static void write_pack_file(void)
display_progress(progress_state, written);
}
- /*
- * Did we write the wrong # entries in the header?
- * If so, rewrite it like in fast-import
- */
if (pack_to_stdout) {
- finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_CLOSE);
+ /*
+ * We never fsync when writing to stdout since we may
+ * not be writing to an actual pack file. For instance,
+ * the upload-pack code passes a pipe here. Calling
+ * fsync on a pipe results in unnecessary
+ * synchronization with the reader on some platforms.
+ */
+ finalize_hashfile(f, hash, FSYNC_COMPONENT_NONE,
+ CSUM_HASH_IN_STREAM | CSUM_CLOSE);
} else if (nr_written == nr_remaining) {
- finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
+ finalize_hashfile(f, hash, FSYNC_COMPONENT_PACK,
+ CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
} else {
- int fd = finalize_hashfile(f, hash, 0);
+ /*
+ * If we wrote the wrong number of entries in the
+ * header, rewrite it like in fast-import.
+ */
+
+ int fd = finalize_hashfile(f, hash, FSYNC_COMPONENT_PACK, 0);
fixup_pack_header_footer(fd, hash, pack_tmp_name,
nr_written, hash, offset);
close(fd);
diff --git a/bulk-checkin.c b/bulk-checkin.c
index 8785b2ac806..a2cf9dcbc8d 100644
--- a/bulk-checkin.c
+++ b/bulk-checkin.c
@@ -53,9 +53,10 @@ static void finish_bulk_checkin(struct bulk_checkin_state *state)
unlink(state->pack_tmp_name);
goto clear_exit;
} else if (state->nr_written == 1) {
- finalize_hashfile(state->f, hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
+ finalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK,
+ CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
} else {
- int fd = finalize_hashfile(state->f, hash, 0);
+ int fd = finalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK, 0);
fixup_pack_header_footer(fd, hash, state->pack_tmp_name,
state->nr_written, hash,
state->offset);
diff --git a/cache.h b/cache.h
index 37a32034b2f..b3cd7d928de 100644
--- a/cache.h
+++ b/cache.h
@@ -993,8 +993,38 @@ void reset_shared_repository(void);
extern int read_replace_refs;
extern char *git_replace_ref_base;
-extern int fsync_object_files;
-extern int use_fsync;
+/*
+ * These values are used to help identify parts of a repository to fsync.
+ * FSYNC_COMPONENT_NONE identifies data that will not be a persistent part of the
+ * repository and so shouldn't be fsynced.
+ */
+enum fsync_component {
+ FSYNC_COMPONENT_NONE,
+ FSYNC_COMPONENT_LOOSE_OBJECT = 1 << 0,
+ FSYNC_COMPONENT_PACK = 1 << 1,
+ FSYNC_COMPONENT_PACK_METADATA = 1 << 2,
+ FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3,
+};
+
+#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \
+ FSYNC_COMPONENT_PACK_METADATA | \
+ FSYNC_COMPONENT_COMMIT_GRAPH)
+
+#define FSYNC_COMPONENTS_OBJECTS (FSYNC_COMPONENT_LOOSE_OBJECT | \
+ FSYNC_COMPONENT_PACK | \
+ FSYNC_COMPONENT_PACK_METADATA | \
+ FSYNC_COMPONENT_COMMIT_GRAPH)
+
+#define FSYNC_COMPONENTS_ALL (FSYNC_COMPONENT_LOOSE_OBJECT | \
+ FSYNC_COMPONENT_PACK | \
+ FSYNC_COMPONENT_PACK_METADATA | \
+ FSYNC_COMPONENT_COMMIT_GRAPH)
+
+
+/*
+ * A bitmask indicating which components of the repo should be fsynced.
+ */
+extern enum fsync_component fsync_components;
enum fsync_method {
FSYNC_METHOD_FSYNC,
@@ -1002,6 +1032,7 @@ enum fsync_method {
};
extern enum fsync_method fsync_method;
+extern int use_fsync;
extern int core_preload_index;
extern int precomposed_unicode;
extern int protect_hfs;
@@ -1757,6 +1788,12 @@ int copy_file_with_time(const char *dst, const char *src, int mode);
void write_or_die(int fd, const void *buf, size_t count);
void fsync_or_die(int fd, const char *);
+inline void fsync_component_or_die(enum fsync_component component, int fd, const char *msg)
+{
+ if (fsync_components & component)
+ fsync_or_die(fd, msg);
+}
+
ssize_t read_in_full(int fd, void *buf, size_t count);
ssize_t write_in_full(int fd, const void *buf, size_t count);
ssize_t pread_in_full(int fd, void *buf, size_t count, off_t offset);
diff --git a/commit-graph.c b/commit-graph.c
index 265c010122e..64897f57d9f 100644
--- a/commit-graph.c
+++ b/commit-graph.c
@@ -1942,7 +1942,8 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)
}
close_commit_graph(ctx->r->objects);
- finalize_hashfile(f, file_hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC);
+ finalize_hashfile(f, file_hash, FSYNC_COMPONENT_COMMIT_GRAPH,
+ CSUM_HASH_IN_STREAM | CSUM_FSYNC);
free_chunkfile(cf);
if (ctx->split) {
diff --git a/config.c b/config.c
index f67f545f839..224563c7b3e 100644
--- a/config.c
+++ b/config.c
@@ -1213,6 +1213,73 @@ static int git_parse_maybe_bool_text(const char *value)
return -1;
}
+static const struct fsync_component_entry {
+ const char *name;
+ enum fsync_component component_bits;
+} fsync_component_table[] = {
+ { "loose-object", FSYNC_COMPONENT_LOOSE_OBJECT },
+ { "pack", FSYNC_COMPONENT_PACK },
+ { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA },
+ { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH },
+ { "objects", FSYNC_COMPONENTS_OBJECTS },
+ { "default", FSYNC_COMPONENTS_DEFAULT },
+ { "all", FSYNC_COMPONENTS_ALL },
+};
+
+static enum fsync_component parse_fsync_components(const char *var, const char *string)
+{
+ enum fsync_component output = 0;
+
+ if (!strcmp(string, "none"))
+ return output;
+
+ while (string) {
+ int i;
+ size_t len;
+ const char *ep;
+ int negated = 0;
+ int found = 0;
+
+ string = string + strspn(string, ", \t\n\r");
+ ep = strchrnul(string, ',');
+ len = ep - string;
+
+ if (*string == '-') {
+ negated = 1;
+ string++;
+ len--;
+ if (!len)
+ warning(_("invalid value for variable %s"), var);
+ }
+
+ if (!len)
+ break;
+
+ for (i = 0; i < ARRAY_SIZE(fsync_component_table); ++i) {
+ const struct fsync_component_entry *entry = &fsync_component_table[i];
+
+ if (strncmp(entry->name, string, len))
+ continue;
+
+ found = 1;
+ if (negated)
+ output &= ~entry->component_bits;
+ else
+ output |= entry->component_bits;
+ }
+
+ if (!found) {
+ char *component = xstrndup(string, len);
+ warning(_("ignoring unknown core.fsync component '%s'"), component);
+ free(component);
+ }
+
+ string = ep;
+ }
+
+ return output;
+}
+
int git_parse_maybe_bool(const char *value)
{
int v = git_parse_maybe_bool_text(value);
@@ -1490,6 +1557,13 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
return 0;
}
+ if (!strcmp(var, "core.fsync")) {
+ if (!value)
+ return config_error_nonbool(var);
+ fsync_components = parse_fsync_components(var, value);
+ return 0;
+ }
+
if (!strcmp(var, "core.fsyncmethod")) {
if (!value)
return config_error_nonbool(var);
@@ -1503,7 +1577,7 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
}
if (!strcmp(var, "core.fsyncobjectfiles")) {
- fsync_object_files = git_config_bool(var, value);
+ warning(_("core.fsyncobjectfiles is deprecated; use core.fsync instead"));
return 0;
}
diff --git a/csum-file.c b/csum-file.c
index 26e8a6df44e..59ef3398ca2 100644
--- a/csum-file.c
+++ b/csum-file.c
@@ -58,7 +58,8 @@ static void free_hashfile(struct hashfile *f)
free(f);
}
-int finalize_hashfile(struct hashfile *f, unsigned char *result, unsigned int flags)
+int finalize_hashfile(struct hashfile *f, unsigned char *result,
+ enum fsync_component component, unsigned int flags)
{
int fd;
@@ -69,7 +70,7 @@ int finalize_hashfile(struct hashfile *f, unsigned char *result, unsigned int fl
if (flags & CSUM_HASH_IN_STREAM)
flush(f, f->buffer, the_hash_algo->rawsz);
if (flags & CSUM_FSYNC)
- fsync_or_die(f->fd, f->name);
+ fsync_component_or_die(component, f->fd, f->name);
if (flags & CSUM_CLOSE) {
if (close(f->fd))
die_errno("%s: sha1 file error on close", f->name);
diff --git a/csum-file.h b/csum-file.h
index 291215b34eb..0d29f528fbc 100644
--- a/csum-file.h
+++ b/csum-file.h
@@ -1,6 +1,7 @@
#ifndef CSUM_FILE_H
#define CSUM_FILE_H
+#include "cache.h"
#include "hash.h"
struct progress;
@@ -38,7 +39,7 @@ int hashfile_truncate(struct hashfile *, struct hashfile_checkpoint *);
struct hashfile *hashfd(int fd, const char *name);
struct hashfile *hashfd_check(const char *name);
struct hashfile *hashfd_throughput(int fd, const char *name, struct progress *tp);
-int finalize_hashfile(struct hashfile *, unsigned char *, unsigned int);
+int finalize_hashfile(struct hashfile *, unsigned char *, enum fsync_component, unsigned int);
void hashwrite(struct hashfile *, const void *, unsigned int);
void hashflush(struct hashfile *f);
void crc32_begin(struct hashfile *);
diff --git a/environment.c b/environment.c
index 3e3620d759f..378424b9af5 100644
--- a/environment.c
+++ b/environment.c
@@ -42,9 +42,9 @@ const char *git_attributes_file;
const char *git_hooks_path;
int zlib_compression_level = Z_BEST_SPEED;
int pack_compression_level = Z_DEFAULT_COMPRESSION;
-int fsync_object_files;
int use_fsync = -1;
enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT;
+enum fsync_component fsync_components = FSYNC_COMPONENTS_DEFAULT;
size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE;
size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT;
size_t delta_base_cache_limit = 96 * 1024 * 1024;
diff --git a/midx.c b/midx.c
index 837b46b2af5..882f91f7d57 100644
--- a/midx.c
+++ b/midx.c
@@ -1406,7 +1406,8 @@ static int write_midx_internal(const char *object_dir,
write_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs);
write_chunkfile(cf, &ctx);
- finalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);
+ finalize_hashfile(f, midx_hash, FSYNC_COMPONENT_PACK_METADATA,
+ CSUM_FSYNC | CSUM_HASH_IN_STREAM);
free_chunkfile(cf);
if (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP))
diff --git a/object-file.c b/object-file.c
index 8be57f48de7..ff29e678204 100644
--- a/object-file.c
+++ b/object-file.c
@@ -1850,8 +1850,7 @@ int hash_object_file(const struct git_hash_algo *algo, const void *buf,
static void close_loose_object(int fd)
{
if (!the_repository->objects->odb->will_destroy) {
- if (fsync_object_files)
- fsync_or_die(fd, "loose object file");
+ fsync_component_or_die(FSYNC_COMPONENT_LOOSE_OBJECT, fd, "loose object file");
}
if (close(fd) != 0)
diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c
index 9c55c1531e1..c16e43d1669 100644
--- a/pack-bitmap-write.c
+++ b/pack-bitmap-write.c
@@ -719,7 +719,8 @@ void bitmap_writer_finish(struct pack_idx_entry **index,
if (options & BITMAP_OPT_HASH_CACHE)
write_hash_cache(f, index, index_nr);
- finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
+ finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA,
+ CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
if (adjust_shared_perm(tmp_file.buf))
die_errno("unable to make temporary bitmap file readable");
diff --git a/pack-write.c b/pack-write.c
index a5846f3a346..51812cb1299 100644
--- a/pack-write.c
+++ b/pack-write.c
@@ -159,9 +159,9 @@ const char *write_idx_file(const char *index_name, struct pack_idx_entry **objec
}
hashwrite(f, sha1, the_hash_algo->rawsz);
- finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE |
- ((opts->flags & WRITE_IDX_VERIFY)
- ? 0 : CSUM_FSYNC));
+ finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA,
+ CSUM_HASH_IN_STREAM | CSUM_CLOSE |
+ ((opts->flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC));
return index_name;
}
@@ -281,8 +281,9 @@ const char *write_rev_file_order(const char *rev_name,
if (rev_name && adjust_shared_perm(rev_name) < 0)
die(_("failed to make %s readable"), rev_name);
- finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE |
- ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC));
+ finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA,
+ CSUM_HASH_IN_STREAM | CSUM_CLOSE |
+ ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC));
return rev_name;
}
@@ -390,7 +391,7 @@ void fixup_pack_header_footer(int pack_fd,
the_hash_algo->final_fn(partial_pack_hash, &old_hash_ctx);
the_hash_algo->final_fn(new_pack_hash, &new_hash_ctx);
write_or_die(pack_fd, new_pack_hash, the_hash_algo->rawsz);
- fsync_or_die(pack_fd, pack_name);
+ fsync_component_or_die(FSYNC_COMPONENT_PACK, pack_fd, pack_name);
}
char *index_pack_lockfile(int ip_out, int *is_well_formed)
diff --git a/read-cache.c b/read-cache.c
index cbe73f14e5e..a0de70195c8 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -3081,7 +3081,7 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile,
return -1;
}
- finalize_hashfile(f, istate->oid.hash, CSUM_HASH_IN_STREAM);
+ finalize_hashfile(f, istate->oid.hash, FSYNC_COMPONENT_NONE, CSUM_HASH_IN_STREAM);
if (close_tempfile_gently(tempfile)) {
error(_("could not close '%s'"), get_tempfile_path(tempfile));
return -1;
--
gitgitgadget
^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v4 3/4] core.fsync: new option to harden the index
2022-02-01 3:33 ` [PATCH v4 " Neeraj K. Singh via GitGitGadget
2022-02-01 3:33 ` [PATCH v4 1/4] core.fsyncmethod: add writeout-only mode Neeraj Singh via GitGitGadget
2022-02-01 3:33 ` [PATCH v4 2/4] core.fsync: introduce granular fsync control Neeraj Singh via GitGitGadget
@ 2022-02-01 3:33 ` Neeraj Singh via GitGitGadget
2022-02-01 3:33 ` [PATCH v4 4/4] core.fsync: add a `derived-metadata` aggregate option Neeraj Singh via GitGitGadget
2022-03-09 23:03 ` [PATCH v5 0/5] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget
4 siblings, 0 replies; 122+ messages in thread
From: Neeraj Singh via GitGitGadget @ 2022-02-01 3:33 UTC (permalink / raw)
To: git
Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps,
Neeraj K. Singh, Neeraj Singh
From: Neeraj Singh <neerajsi@microsoft.com>
This commit introduces the new ability for the user to harden
the index. In the event of a system crash, the index must be
durable for the user to actually find a file that has been added
to the repo and then deleted from the working tree.
We use the presence of the COMMIT_LOCK flag and absence of the
alternate_index_output as a proxy for determining whether we're
updating the persistent index of the repo or some temporary
index. We don't sync these temporary indexes.
Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
---
Documentation/config/core.txt | 1 +
cache.h | 4 +++-
config.c | 1 +
read-cache.c | 19 +++++++++++++------
4 files changed, 18 insertions(+), 7 deletions(-)
diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
index 4f1747ec871..8e5b7a795ab 100644
--- a/Documentation/config/core.txt
+++ b/Documentation/config/core.txt
@@ -561,6 +561,7 @@ core.fsync::
* `pack` hardens objects added to the repo in packfile form.
* `pack-metadata` hardens packfile bitmaps and indexes.
* `commit-graph` hardens the commit graph file.
+* `index` hardens the index when it is modified.
* `objects` is an aggregate option that includes `loose-objects`, `pack`,
`pack-metadata`, and `commit-graph`.
* `default` is an aggregate option that is equivalent to `objects,-loose-object`
diff --git a/cache.h b/cache.h
index b3cd7d928de..9f3c1ec4c42 100644
--- a/cache.h
+++ b/cache.h
@@ -1004,6 +1004,7 @@ enum fsync_component {
FSYNC_COMPONENT_PACK = 1 << 1,
FSYNC_COMPONENT_PACK_METADATA = 1 << 2,
FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3,
+ FSYNC_COMPONENT_INDEX = 1 << 4,
};
#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \
@@ -1018,7 +1019,8 @@ enum fsync_component {
#define FSYNC_COMPONENTS_ALL (FSYNC_COMPONENT_LOOSE_OBJECT | \
FSYNC_COMPONENT_PACK | \
FSYNC_COMPONENT_PACK_METADATA | \
- FSYNC_COMPONENT_COMMIT_GRAPH)
+ FSYNC_COMPONENT_COMMIT_GRAPH | \
+ FSYNC_COMPONENT_INDEX)
/*
diff --git a/config.c b/config.c
index 224563c7b3e..325644e3c2c 100644
--- a/config.c
+++ b/config.c
@@ -1221,6 +1221,7 @@ static const struct fsync_component_entry {
{ "pack", FSYNC_COMPONENT_PACK },
{ "pack-metadata", FSYNC_COMPONENT_PACK_METADATA },
{ "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH },
+ { "index", FSYNC_COMPONENT_INDEX },
{ "objects", FSYNC_COMPONENTS_OBJECTS },
{ "default", FSYNC_COMPONENTS_DEFAULT },
{ "all", FSYNC_COMPONENTS_ALL },
diff --git a/read-cache.c b/read-cache.c
index a0de70195c8..eb02439ab4b 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -2837,7 +2837,7 @@ static int record_ieot(void)
* rely on it.
*/
static int do_write_index(struct index_state *istate, struct tempfile *tempfile,
- int strip_extensions)
+ int strip_extensions, unsigned flags)
{
uint64_t start = getnanotime();
struct hashfile *f;
@@ -2851,6 +2851,7 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile,
struct strbuf previous_name_buf = STRBUF_INIT, *previous_name;
int drop_cache_tree = istate->drop_cache_tree;
off_t offset;
+ int csum_fsync_flag;
int ieot_entries = 1;
struct index_entry_offset_table *ieot = NULL;
int nr, nr_threads;
@@ -3081,7 +3082,13 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile,
return -1;
}
- finalize_hashfile(f, istate->oid.hash, FSYNC_COMPONENT_NONE, CSUM_HASH_IN_STREAM);
+ csum_fsync_flag = 0;
+ if (!alternate_index_output && (flags & COMMIT_LOCK))
+ csum_fsync_flag = CSUM_FSYNC;
+
+ finalize_hashfile(f, istate->oid.hash, FSYNC_COMPONENT_INDEX,
+ CSUM_HASH_IN_STREAM | csum_fsync_flag);
+
if (close_tempfile_gently(tempfile)) {
error(_("could not close '%s'"), get_tempfile_path(tempfile));
return -1;
@@ -3136,7 +3143,7 @@ static int do_write_locked_index(struct index_state *istate, struct lock_file *l
*/
trace2_region_enter_printf("index", "do_write_index", the_repository,
"%s", get_lock_file_path(lock));
- ret = do_write_index(istate, lock->tempfile, 0);
+ ret = do_write_index(istate, lock->tempfile, 0, flags);
trace2_region_leave_printf("index", "do_write_index", the_repository,
"%s", get_lock_file_path(lock));
@@ -3230,7 +3237,7 @@ static int clean_shared_index_files(const char *current_hex)
}
static int write_shared_index(struct index_state *istate,
- struct tempfile **temp)
+ struct tempfile **temp, unsigned flags)
{
struct split_index *si = istate->split_index;
int ret, was_full = !istate->sparse_index;
@@ -3240,7 +3247,7 @@ static int write_shared_index(struct index_state *istate,
trace2_region_enter_printf("index", "shared/do_write_index",
the_repository, "%s", get_tempfile_path(*temp));
- ret = do_write_index(si->base, *temp, 1);
+ ret = do_write_index(si->base, *temp, 1, flags);
trace2_region_leave_printf("index", "shared/do_write_index",
the_repository, "%s", get_tempfile_path(*temp));
@@ -3349,7 +3356,7 @@ int write_locked_index(struct index_state *istate, struct lock_file *lock,
ret = do_write_locked_index(istate, lock, flags);
goto out;
}
- ret = write_shared_index(istate, &temp);
+ ret = write_shared_index(istate, &temp, flags);
saved_errno = errno;
if (is_tempfile_active(temp))
--
gitgitgadget
^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v4 4/4] core.fsync: add a `derived-metadata` aggregate option
2022-02-01 3:33 ` [PATCH v4 " Neeraj K. Singh via GitGitGadget
` (2 preceding siblings ...)
2022-02-01 3:33 ` [PATCH v4 3/4] core.fsync: new option to harden the index Neeraj Singh via GitGitGadget
@ 2022-02-01 3:33 ` Neeraj Singh via GitGitGadget
2022-03-09 23:03 ` [PATCH v5 0/5] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget
4 siblings, 0 replies; 122+ messages in thread
From: Neeraj Singh via GitGitGadget @ 2022-02-01 3:33 UTC (permalink / raw)
To: git
Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps,
Neeraj K. Singh, Neeraj Singh
From: Neeraj Singh <neerajsi@microsoft.com>
This commit adds an aggregate option that currently includes the
commit-graph file and pack metadata (indexes and bitmaps).
The user may want to exclude this set from durability since they can be
recomputed from other data if they wind up corrupt or missing.
This is split out from the other patches in the series since it is
an optional nice-to-have that might be controversial.
Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
---
Documentation/config/core.txt | 6 +++---
cache.h | 7 ++++---
config.c | 1 +
3 files changed, 8 insertions(+), 6 deletions(-)
diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
index 8e5b7a795ab..21092f3a4d1 100644
--- a/Documentation/config/core.txt
+++ b/Documentation/config/core.txt
@@ -562,9 +562,9 @@ core.fsync::
* `pack-metadata` hardens packfile bitmaps and indexes.
* `commit-graph` hardens the commit graph file.
* `index` hardens the index when it is modified.
-* `objects` is an aggregate option that includes `loose-objects`, `pack`,
- `pack-metadata`, and `commit-graph`.
-* `default` is an aggregate option that is equivalent to `objects,-loose-object`
+* `objects` is an aggregate option that includes `loose-objects` and `pack`.
+* `derived-metadata` is an aggregate option that includes `pack-metadata` and `commit-graph`.
+* `default` is an aggregate option that is equivalent to `objects,derived-metadata,-loose-object`
* `all` is an aggregate option that syncs all individual components above.
core.fsyncMethod::
diff --git a/cache.h b/cache.h
index 9f3c1ec4c42..3327cf6af0b 100644
--- a/cache.h
+++ b/cache.h
@@ -1012,9 +1012,10 @@ enum fsync_component {
FSYNC_COMPONENT_COMMIT_GRAPH)
#define FSYNC_COMPONENTS_OBJECTS (FSYNC_COMPONENT_LOOSE_OBJECT | \
- FSYNC_COMPONENT_PACK | \
- FSYNC_COMPONENT_PACK_METADATA | \
- FSYNC_COMPONENT_COMMIT_GRAPH)
+ FSYNC_COMPONENT_PACK)
+
+#define FSYNC_COMPONENTS_DERIVED_METADATA (FSYNC_COMPONENT_PACK_METADATA | \
+ FSYNC_COMPONENT_COMMIT_GRAPH)
#define FSYNC_COMPONENTS_ALL (FSYNC_COMPONENT_LOOSE_OBJECT | \
FSYNC_COMPONENT_PACK | \
diff --git a/config.c b/config.c
index 325644e3c2c..64a8a4d7c2a 100644
--- a/config.c
+++ b/config.c
@@ -1223,6 +1223,7 @@ static const struct fsync_component_entry {
{ "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH },
{ "index", FSYNC_COMPONENT_INDEX },
{ "objects", FSYNC_COMPONENTS_OBJECTS },
+ { "derived-metadata", FSYNC_COMPONENTS_DERIVED_METADATA },
{ "default", FSYNC_COMPONENTS_DEFAULT },
{ "all", FSYNC_COMPONENTS_ALL },
};
--
gitgitgadget
^ permalink raw reply related [flat|nested] 122+ messages in thread
* Re: [PATCH v4 2/4] core.fsync: introduce granular fsync control
2022-02-01 3:33 ` [PATCH v4 2/4] core.fsync: introduce granular fsync control Neeraj Singh via GitGitGadget
@ 2022-02-02 0:51 ` Junio C Hamano
2022-02-02 1:42 ` Junio C Hamano
2022-02-11 20:38 ` Neeraj Singh
0 siblings, 2 replies; 122+ messages in thread
From: Junio C Hamano @ 2022-02-02 0:51 UTC (permalink / raw)
To: Neeraj Singh via GitGitGadget
Cc: git, rsbecker, bagasdotme, newren, avarab, nksingh85, ps,
Neeraj K. Singh
"Neeraj Singh via GitGitGadget" <gitgitgadget@gmail.com> writes:
> +core.fsync::
> + A comma-separated list of parts of the repository which should be
> + hardened via the core.fsyncMethod when created or modified. You can
> + disable hardening of any component by prefixing it with a '-'. Later
> + items take precedence over earlier ones in the list. For example,
> + `core.fsync=all,-pack-metadata` means "harden everything except pack
> + metadata." Items that are not hardened may be lost in the event of an
> + unclean system shutdown.
> ++
> +* `none` disables fsync completely. This must be specified alone.
> +* `loose-object` hardens objects added to the repo in loose-object form.
> +* `pack` hardens objects added to the repo in packfile form.
> +* `pack-metadata` hardens packfile bitmaps and indexes.
> +* `commit-graph` hardens the commit graph file.
> +* `objects` is an aggregate option that includes `loose-objects`, `pack`,
> + `pack-metadata`, and `commit-graph`.
> +* `default` is an aggregate option that is equivalent to `objects,-loose-object`
> +* `all` is an aggregate option that syncs all individual components above.
I am not quite sure if this is way too complex (e.g. what does it
mean that we do not care much about loose-object safety while we do
care about commit-graph files?) and at the same time it is too
limited (e.g. if it makes sense to say a class of items deserve more
protection than another class of items, don't we want to be able to
say "class X is ultra-precious so use method A on them, while class
Y is mildly precious and use method B on them, everything else are
not that important and doing the default thing is just fine").
If we wanted to allow the "matrix" kind of flexibility, I think the
way to do so would be
fsync.<class>.method = <value>
e.g.
[fsync "default"] method = none
[fsync "loose-object"] method = fsync
[fsync "pack-metadata"] method = writeout-only
Where do we expect users to take the core.fsync settings from? Per
repository? If it is from per user (i.e. $HOME/.gitconfig), do
people tend to share it across systems (not necessarily over NFS)
with the same contents? If so, I am not sure if fsync.method that
is way too close to the actual "implementation" is a good idea to
begin with. From end-user's point of view, it may be easier to
express "class X is ultra-precious, and class Y and Z are mildly
so", with something like fsync.<class>.level = <how-precious> and
let the Git implementation on each platform choose the appropriate
fsync method to protect the stuff at that precious-ness.
Thanks.
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v4 2/4] core.fsync: introduce granular fsync control
2022-02-02 0:51 ` Junio C Hamano
@ 2022-02-02 1:42 ` Junio C Hamano
2022-02-11 21:18 ` Neeraj Singh
2022-02-11 20:38 ` Neeraj Singh
1 sibling, 1 reply; 122+ messages in thread
From: Junio C Hamano @ 2022-02-02 1:42 UTC (permalink / raw)
To: Neeraj Singh via GitGitGadget
Cc: git, rsbecker, bagasdotme, newren, avarab, nksingh85, ps,
Neeraj K. Singh
Junio C Hamano <gitster@pobox.com> writes:
> I am not quite sure if this is way too complex (e.g. what does it
> mean that we do not care much about loose-object safety while we do
> care about commit-graph files?) and at the same time it is too
> limited (e.g. if it makes sense to say a class of items deserve more
> protection than another class of items, don't we want to be able to
> say "class X is ultra-precious so use method A on them, while class
> Y is mildly precious and use method B on them, everything else are
> not that important and doing the default thing is just fine").
>
> If we wanted to allow the "matrix" kind of flexibility,...
To continue with the thinking aloud...
Sometimes configuration flexibility is truly needed, but often it is
just a sign of designer being lazy and not thinking it through as an
end-user facing problem. In other words, "I am giving enough knobs
to you, so it is up to you to express your policy in whatever way
you want with the knobs provided" is a very irresponsible thing to
tell end-users.
And this one smells like the case of a lazy design.
It may be that it makes sense in some workflows to protect
commit-graph files less than object files and pack.idx files can be
corrupted as long as pack.pack files are adequately protected
because the former can be recomputed from the latter, but in no
workflows, the reverse would be true. Yet the design gives such
needless flexibility, which makes it hard for lay end-users to
choose the best combination and allows them to protect .idx files
more than .pack files by mistake, for example.
I am wondering if the classification itself introduced by this step
actually can form a natural and linear progression of safe-ness. By
default, we'd want _all_ classes of things to be equally safe, but
at one level down, there is "protect things that are not
recomputable, but recomputable things can be left to the system"
level, and there would be even riskier "protect packs as it would
hurt a _lot_ to lose them, but losing loose ones will typically lose
only the most recent work, and they are less valuable" level.
If we, as the Git experts, spend extra brain cycles to come up with
an easy to understand spectrum of performance vs durability
trade-off, end-users won't have to learn the full flexibility and
easily take the advice from experts. They just need to say what
level of durability they want (or how much durability they can risk
in exchange for an additional throughput), and leave the rest to us.
On the core.fsyncMethod side, the same suggestion applies.
Once we know the desired level of performance vs durability
trade-off from the user, we, as the impolementors, should know the
best method, for each class of items, to achieve that durability on
each platform when writing it to the storage, without exposing the
low level details of the implementation that only the Git folks need
to be aware of.
So, from the end-user UI perspective, I'd very much prefer if we can
just come up with a single scalar variable, (say "fsync.durability"
that ranges from "conservative" to "performance") that lets our
users express the level of durability desired. The combination of
core.fsyncMethod and core.fsync are one variable too many, and the
latter being a variable that takes a list of things as its value
makes it even worse to sell to the end users.
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v4 2/4] core.fsync: introduce granular fsync control
2022-02-02 0:51 ` Junio C Hamano
2022-02-02 1:42 ` Junio C Hamano
@ 2022-02-11 20:38 ` Neeraj Singh
1 sibling, 0 replies; 122+ messages in thread
From: Neeraj Singh @ 2022-02-11 20:38 UTC (permalink / raw)
To: Junio C Hamano
Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker,
Bagas Sanjaya, Elijah Newren,
Ævar Arnfjörð Bjarmason, Patrick Steinhardt,
Neeraj K. Singh
Apologies in advance for the delayed reply. I've finally been able to
return to Git after an absence.
On Tue, Feb 1, 2022 at 4:51 PM Junio C Hamano <gitster@pobox.com> wrote:
>
> "Neeraj Singh via GitGitGadget" <gitgitgadget@gmail.com> writes:
>
> > +core.fsync::
> > + A comma-separated list of parts of the repository which should be
> > + hardened via the core.fsyncMethod when created or modified. You can
> > + disable hardening of any component by prefixing it with a '-'. Later
> > + items take precedence over earlier ones in the list. For example,
> > + `core.fsync=all,-pack-metadata` means "harden everything except pack
> > + metadata." Items that are not hardened may be lost in the event of an
> > + unclean system shutdown.
> > ++
> > +* `none` disables fsync completely. This must be specified alone.
> > +* `loose-object` hardens objects added to the repo in loose-object form.
> > +* `pack` hardens objects added to the repo in packfile form.
> > +* `pack-metadata` hardens packfile bitmaps and indexes.
> > +* `commit-graph` hardens the commit graph file.
> > +* `objects` is an aggregate option that includes `loose-objects`, `pack`,
> > + `pack-metadata`, and `commit-graph`.
> > +* `default` is an aggregate option that is equivalent to `objects,-loose-object`
> > +* `all` is an aggregate option that syncs all individual components above.
>
> I am not quite sure if this is way too complex (e.g. what does it
> mean that we do not care much about loose-object safety while we do
> care about commit-graph files?) and at the same time it is too
> limited (e.g. if it makes sense to say a class of items deserve more
> protection than another class of items, don't we want to be able to
> say "class X is ultra-precious so use method A on them, while class
> Y is mildly precious and use method B on them, everything else are
> not that important and doing the default thing is just fine").
>
> If we wanted to allow the "matrix" kind of flexibility, I think the
> way to do so would be
>
> fsync.<class>.method = <value>
>
> e.g.
>
> [fsync "default"] method = none
> [fsync "loose-object"] method = fsync
> [fsync "pack-metadata"] method = writeout-only
>
I don't believe it makes sense to offer a full matrix of what to fsync
and what method to use, since the method is a property of the
filesystem and OS the repo is running on, while the list of things to
fsync is more a selection of what the user values. So if I'm hosting
on APFS on macOS or NTFS on Windows, I'd want to set the fsyncMethod
to batch so that I can get good performance at the safety level I
choose. If I'm working on my maintainer repo, I'd maybe not want to
fsync anything, but I'd want to fsync everything when working on my
developer repo.
> Where do we expect users to take the core.fsync settings from? Per
> repository? If it is from per user (i.e. $HOME/.gitconfig), do
> people tend to share it across systems (not necessarily over NFS)
> with the same contents? If so, I am not sure if fsync.method that
> is way too close to the actual "implementation" is a good idea to
> begin with. From end-user's point of view, it may be easier to
> express "class X is ultra-precious, and class Y and Z are mildly
> so", with something like fsync.<class>.level = <how-precious> and
> let the Git implementation on each platform choose the appropriate
> fsync method to protect the stuff at that precious-ness.
>
I expect the vast majority of users to have whatever setting is baked
into their build of Git. For the users that want to do something
different, I expect them to have core.fsyncMethod and core.fsync
configured per-user for the majority of their repos. Some repos might
have custom settings that override the per-user settings: 1) Ephemeral
repos that don't contain unique data would probably want to set
core.fsync=none. 2) Repos hosting on NFS or on a different FS may have
a stricter core.fsyncmethod setting.
(More more text to follow in reply to your next email).
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v4 2/4] core.fsync: introduce granular fsync control
2022-02-02 1:42 ` Junio C Hamano
@ 2022-02-11 21:18 ` Neeraj Singh
2022-02-11 22:19 ` Junio C Hamano
0 siblings, 1 reply; 122+ messages in thread
From: Neeraj Singh @ 2022-02-11 21:18 UTC (permalink / raw)
To: Junio C Hamano
Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker,
Bagas Sanjaya, Elijah Newren,
Ævar Arnfjörð Bjarmason, Patrick Steinhardt,
Neeraj K. Singh
On Tue, Feb 1, 2022 at 5:42 PM Junio C Hamano <gitster@pobox.com> wrote:
>
> Junio C Hamano <gitster@pobox.com> writes:
>
> > I am not quite sure if this is way too complex (e.g. what does it
> > mean that we do not care much about loose-object safety while we do
> > care about commit-graph files?) and at the same time it is too
> > limited (e.g. if it makes sense to say a class of items deserve more
> > protection than another class of items, don't we want to be able to
> > say "class X is ultra-precious so use method A on them, while class
> > Y is mildly precious and use method B on them, everything else are
> > not that important and doing the default thing is just fine").
> >
> > If we wanted to allow the "matrix" kind of flexibility,...
>
> To continue with the thinking aloud...
>
> Sometimes configuration flexibility is truly needed, but often it is
> just a sign of designer being lazy and not thinking it through as an
> end-user facing problem. In other words, "I am giving enough knobs
> to you, so it is up to you to express your policy in whatever way
> you want with the knobs provided" is a very irresponsible thing to
> tell end-users.
>
> And this one smells like the case of a lazy design.
>
> It may be that it makes sense in some workflows to protect
> commit-graph files less than object files and pack.idx files can be
> corrupted as long as pack.pack files are adequately protected
> because the former can be recomputed from the latter, but in no
> workflows, the reverse would be true. Yet the design gives such
> needless flexibility, which makes it hard for lay end-users to
> choose the best combination and allows them to protect .idx files
> more than .pack files by mistake, for example.
>
> I am wondering if the classification itself introduced by this step
> actually can form a natural and linear progression of safe-ness. By
> default, we'd want _all_ classes of things to be equally safe, but
> at one level down, there is "protect things that are not
> recomputable, but recomputable things can be left to the system"
> level, and there would be even riskier "protect packs as it would
> hurt a _lot_ to lose them, but losing loose ones will typically lose
> only the most recent work, and they are less valuable" level.
>
> If we, as the Git experts, spend extra brain cycles to come up with
> an easy to understand spectrum of performance vs durability
> trade-off, end-users won't have to learn the full flexibility and
> easily take the advice from experts. They just need to say what
> level of durability they want (or how much durability they can risk
> in exchange for an additional throughput), and leave the rest to us.
>
> On the core.fsyncMethod side, the same suggestion applies.
>
> Once we know the desired level of performance vs durability
> trade-off from the user, we, as the impolementors, should know the
> best method, for each class of items, to achieve that durability on
> each platform when writing it to the storage, without exposing the
> low level details of the implementation that only the Git folks need
> to be aware of.
>
> So, from the end-user UI perspective, I'd very much prefer if we can
> just come up with a single scalar variable, (say "fsync.durability"
> that ranges from "conservative" to "performance") that lets our
> users express the level of durability desired. The combination of
> core.fsyncMethod and core.fsync are one variable too many, and the
> latter being a variable that takes a list of things as its value
> makes it even worse to sell to the end users.
I see the value in simplifying the core.fsync configuration to a
single scalar knob of preciousness. The main motivation for this more
granular scheme is that I didn't think the current configuration
follows a sensible principle. We should be fsyncing the loose objects,
index, refs, and config files in addition to what we're already
syncing today. On macOS, we should be doing a full hardware flush if
we've said we want to fsync. But you expressed the notion in [1] that
we don't want to degrade the performance of the vast majority of users
who are happy with the current "unprincipled but mostly works"
configuration. I agree with that sentiment, but it leads to a design
where we can express the current configuration, which does not follow
a scalar hierarchy.
The aggregate core.fsync options are meant to provide a way for us to
recommend a sensible configuration to the user without having to get
into the intricacies of repo layout. Maybe we can define and document
aggregate options that make sense in terms of a scalar level of
preciousness.
One reason to keep core.fsyncMethod separate from the core.fsync knob
is that it's more a property of the system and FS the repo is on,
rather than the user's value of the repo. We could try to auto-detect
some known filesystems that might support batch mode using 'statfs',
but having a table like that in Git would go out of date over time.
Please let me know what you think about these justifications for the
current design. I'd be happy to make a change if the current
constraint of "keep the default config the same" can be relaxed in
some way. I'd also be happy to go back to some variation of
expressing 'core.fsyncObjectFiles = batch' and leaving the rest of
fsync story alone.
Thanks,
Neeraj
[1] https://lore.kernel.org/git/xmqqtuilyfls.fsf@gitster.g/
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v4 2/4] core.fsync: introduce granular fsync control
2022-02-11 21:18 ` Neeraj Singh
@ 2022-02-11 22:19 ` Junio C Hamano
2022-02-11 23:04 ` Neeraj Singh
0 siblings, 1 reply; 122+ messages in thread
From: Junio C Hamano @ 2022-02-11 22:19 UTC (permalink / raw)
To: Neeraj Singh
Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker,
Bagas Sanjaya, Elijah Newren,
Ævar Arnfjörð Bjarmason, Patrick Steinhardt,
Neeraj K. Singh
Neeraj Singh <nksingh85@gmail.com> writes:
> The main motivation for this more granular scheme is that I didn't
> think the current configuration follows a sensible principle. We
> should be fsyncing the loose objects, index, refs, and config
> files in addition to what we're already syncing today. On macOS,
> we should be doing a full hardware flush if we've said we want to
> fsync.
If the "robustness vs performance" trade-off is unevenly made in the
current code, then that is a very good problem to address first, and
such a change is very much justified on its own.
Perhaps "this is not a primary work repository but is used only to
follow external site to build, hence no fsync is fine" folks, who do
not have core.fsyncObjectFiles set to true, may appreciate if we
stopped doing fsync for packs and other things. As the Boolean
core.fsyncObjectFiles is the only end-user visible knob to express
how the end-users express the trade-off, a good first step would be
to align other file accesses to the preference expressed by it, i.e.
others who say they want fsync in the current scheme would
appreciate if we start fsync in places like ref-files backend.
Making the choice more granular, from Boolean "yes/no", to linear
levels, would also be a good idea. Doing both at the same time may
make it harder to explain and justify, but as long as at the end, if
"very performant" choice uniformly does not do any fsync while
"ultra durable" choice makes a uniform effort across subsystems to
make sure bits hit the platter, it would be a very good idea to do
them.
Thanks.
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v4 2/4] core.fsync: introduce granular fsync control
2022-02-11 22:19 ` Junio C Hamano
@ 2022-02-11 23:04 ` Neeraj Singh
2022-02-11 23:15 ` Junio C Hamano
0 siblings, 1 reply; 122+ messages in thread
From: Neeraj Singh @ 2022-02-11 23:04 UTC (permalink / raw)
To: Junio C Hamano
Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker,
Bagas Sanjaya, Elijah Newren,
Ævar Arnfjörð Bjarmason, Patrick Steinhardt,
Neeraj K. Singh
On Fri, Feb 11, 2022 at 2:19 PM Junio C Hamano <gitster@pobox.com> wrote:
>
> Neeraj Singh <nksingh85@gmail.com> writes:
>
> > The main motivation for this more granular scheme is that I didn't
> > think the current configuration follows a sensible principle. We
> > should be fsyncing the loose objects, index, refs, and config
> > files in addition to what we're already syncing today. On macOS,
> > we should be doing a full hardware flush if we've said we want to
> > fsync.
>
> If the "robustness vs performance" trade-off is unevenly made in the
> current code, then that is a very good problem to address first, and
> such a change is very much justified on its own.
>
> Perhaps "this is not a primary work repository but is used only to
> follow external site to build, hence no fsync is fine" folks, who do
> not have core.fsyncObjectFiles set to true, may appreciate if we
> stopped doing fsync for packs and other things. As the Boolean
> core.fsyncObjectFiles is the only end-user visible knob to express
> how the end-users express the trade-off, a good first step would be
> to align other file accesses to the preference expressed by it, i.e.
> others who say they want fsync in the current scheme would
> appreciate if we start fsync in places like ref-files backend.
>
In practice, almost all users have core.fsyncObjectFiles set to the
platform default, which is 'false' everywhere besides Windows. So at
minimum, we have to take default to mean that we maintain behavior no
weaker than the current version of Git, otherwise users will start
losing their data. The advantage of introducing a new knob is that we
don't have to try to divine why the user set a particular value of
core.fsyncObjectFiles.
> Making the choice more granular, from Boolean "yes/no", to linear
> levels, would also be a good idea. Doing both at the same time may
> make it harder to explain and justify, but as long as at the end, if
> "very performant" choice uniformly does not do any fsync while
> "ultra durable" choice makes a uniform effort across subsystems to
> make sure bits hit the platter, it would be a very good idea to do
> them.
>
One path to get to your suggestion from the current patch series would
be to remove the component-specific options and only provide aggregate
options. Alternatively, we could just not document the
component-specific options and leave them available to be people who
read source code. So if I rename the aggregate options in terms of
'levels of durability', and only document those, would that be
acceptable?
Thanks for the review!
-Neeraj
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v4 2/4] core.fsync: introduce granular fsync control
2022-02-11 23:04 ` Neeraj Singh
@ 2022-02-11 23:15 ` Junio C Hamano
2022-02-12 0:39 ` rsbecker
2022-02-14 7:04 ` Patrick Steinhardt
0 siblings, 2 replies; 122+ messages in thread
From: Junio C Hamano @ 2022-02-11 23:15 UTC (permalink / raw)
To: Neeraj Singh
Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker,
Bagas Sanjaya, Elijah Newren,
Ævar Arnfjörð Bjarmason, Patrick Steinhardt,
Neeraj K. Singh
Neeraj Singh <nksingh85@gmail.com> writes:
> In practice, almost all users have core.fsyncObjectFiles set to the
> platform default, which is 'false' everywhere besides Windows. So at
> minimum, we have to take default to mean that we maintain behavior no
> weaker than the current version of Git, otherwise users will start
> losing their data.
Would they? If they think platform default is performant and safe
enough for their use, as long as our adjustment is out outrageously
more dangerous or less performant, I do not think "no weaker than"
is a strict requirement. If we were overly conservative in some
areas than the "platform default", making it less conservative in
those areas to match the looseness of other areas should be OK and
vice versa.
> One path to get to your suggestion from the current patch series would
> be to remove the component-specific options and only provide aggregate
> options. Alternatively, we could just not document the
> component-specific options and leave them available to be people who
> read source code. So if I rename the aggregate options in terms of
> 'levels of durability', and only document those, would that be
> acceptable?
In any case, if others who reviewed the series in the past are happy
with the "two knobs" approach and are willing to jump in to help new
users who will be confused with one knob too many, I actually am OK
with the series that I called "overly complex". Let me let them
weigh in before I can answer that question.
Thanks.
^ permalink raw reply [flat|nested] 122+ messages in thread
* RE: [PATCH v4 2/4] core.fsync: introduce granular fsync control
2022-02-11 23:15 ` Junio C Hamano
@ 2022-02-12 0:39 ` rsbecker
2022-02-14 7:04 ` Patrick Steinhardt
1 sibling, 0 replies; 122+ messages in thread
From: rsbecker @ 2022-02-12 0:39 UTC (permalink / raw)
To: 'Junio C Hamano', 'Neeraj Singh'
Cc: 'Neeraj Singh via GitGitGadget', 'Git List',
'Bagas Sanjaya', 'Elijah Newren',
'Ævar Arnfjörð Bjarmason',
'Patrick Steinhardt', 'Neeraj K. Singh'
On February 11, 2022 6:15 PM, Junio C Hamano wrote:
> Neeraj Singh <nksingh85@gmail.com> writes:
>
> > In practice, almost all users have core.fsyncObjectFiles set to the
> > platform default, which is 'false' everywhere besides Windows. So at
> > minimum, we have to take default to mean that we maintain behavior no
> > weaker than the current version of Git, otherwise users will start
> > losing their data.
>
> Would they? If they think platform default is performant and safe
> enough for their use, as long as our adjustment is out outrageously more
> dangerous or less performant, I do not think "no weaker than"
> is a strict requirement. If we were overly conservative in some areas than the
> "platform default", making it less conservative in those areas to match the
> looseness of other areas should be OK and vice versa.
>
> > One path to get to your suggestion from the current patch series would
> > be to remove the component-specific options and only provide aggregate
> > options. Alternatively, we could just not document the
> > component-specific options and leave them available to be people who
> > read source code. So if I rename the aggregate options in terms of
> > 'levels of durability', and only document those, would that be
> > acceptable?
>
> In any case, if others who reviewed the series in the past are happy with the
> "two knobs" approach and are willing to jump in to help new users who will be
> confused with one knob too many, I actually am OK with the series that I called
> "overly complex". Let me let them weigh in before I can answer that question.
On behalf of those who are likely to set fsync to true in all cases, because a SIGSEGV or some other early abort will cause changes to be lost, I am not happy with excessive knobs, as we will have to ensure that the are all set to "write this out to disk as quickly as soon as possible or else". I end up having to teach large numbers of people about these settings and think that excessive controls in this area are overwhelmingly bad. This is not a "windows vs. everything else" situation. Not all platforms write buffered by default. fwrite and write behave different on NonStop (and other POSIXy things, and some variants are fully virtualized, so who knows what the hypervisor will do - it's bad enough that there is even an option to tell the hypervisor to keep things in memory until convenient to write even on RAID 0/1 emulation). Teaching people how to use the knobs correctly is going to be a challenge. For me, I'm always willing to sacrifice performance for reliability - 100% of the time. I'm sure this whole series is going to be problematic. I'm sorry but that's my position on it. The default position of all knobs must be to force the write and keeping it simple so that variations on knob settings give different results is not going to end well.
Sincerely,
Randall
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v4 2/4] core.fsync: introduce granular fsync control
2022-02-11 23:15 ` Junio C Hamano
2022-02-12 0:39 ` rsbecker
@ 2022-02-14 7:04 ` Patrick Steinhardt
2022-02-14 17:17 ` Junio C Hamano
1 sibling, 1 reply; 122+ messages in thread
From: Patrick Steinhardt @ 2022-02-14 7:04 UTC (permalink / raw)
To: Junio C Hamano
Cc: Neeraj Singh, Neeraj Singh via GitGitGadget, Git List,
Randall S. Becker, Bagas Sanjaya, Elijah Newren,
Ævar Arnfjörð Bjarmason, Neeraj K. Singh
[-- Attachment #1: Type: text/plain, Size: 4425 bytes --]
On Fri, Feb 11, 2022 at 03:15:20PM -0800, Junio C Hamano wrote:
> Neeraj Singh <nksingh85@gmail.com> writes:
>
> > In practice, almost all users have core.fsyncObjectFiles set to the
> > platform default, which is 'false' everywhere besides Windows. So at
> > minimum, we have to take default to mean that we maintain behavior no
> > weaker than the current version of Git, otherwise users will start
> > losing their data.
>
> Would they? If they think platform default is performant and safe
> enough for their use, as long as our adjustment is out outrageously
> more dangerous or less performant, I do not think "no weaker than"
> is a strict requirement. If we were overly conservative in some
> areas than the "platform default", making it less conservative in
> those areas to match the looseness of other areas should be OK and
> vice versa.
>
> > One path to get to your suggestion from the current patch series would
> > be to remove the component-specific options and only provide aggregate
> > options. Alternatively, we could just not document the
> > component-specific options and leave them available to be people who
> > read source code. So if I rename the aggregate options in terms of
> > 'levels of durability', and only document those, would that be
> > acceptable?
>
> In any case, if others who reviewed the series in the past are happy
> with the "two knobs" approach and are willing to jump in to help new
> users who will be confused with one knob too many, I actually am OK
> with the series that I called "overly complex". Let me let them
> weigh in before I can answer that question.
>
> Thanks.
I wonder whether it makes sense to distinguish client- and server-side
requirements. While we probably want to "do the right thing" on the
client-side by default so that we don't have to teach users that "We may
use data in some cases on some systems, but not in other cases on other
systems by default." On the server-side folks who implement the Git
hosting are typically a lot more knowledgeable with regards to how Git
behaves, and it can be expected of them to dig a lot deeper than we can
and should reasonably expect from a user.
One point I really care about and that I think is extremely important in
the context of both client and server is data consistency. While it is
bad to lose data, the reality is that it can simply happen when systems
hit exceptional cases like a hard-reset. But what we really must not let
happen is that as a result, a repository gets corrupted because of this
hard reset. In the context of client-side repositories a user will be
left to wonder how to fix the repository, while on the server-side we
need to go in and somehow repair the repository fast or otherwise users
will come to us and complain their repository stopped working.
Our current defaults can end up with repository corruption though. We
don't sync object files before renaming them into place by default, and
furthermore we don't even give a knob to fsync loose references before
renaming them. On gitlab.com we enable "core.fsyncObjectFiles" and thus
to the best of my knowledge never hit a corrupted ODB until now. But
what we do regularly hit is corrupted references because there is no
knob to fsync them.
Furthermore, I really think that the advice in git-config(1) with
regards to loose syncing loose objects that "This is a total waste of
time and effort" is wrong. Filesystem developers have repeatedly stated
that for proper atomicity guarantees, a file must be synced to disk
before renaming it into place [1][2][3]. So from the perspective of the
people who write Linux-based filesystems our application is "broken"
right now, even though there are mechanisms in place which try to work
around our brokenness by using heuristics to detect what we're doing.
To summarize my take: while the degree of durability may be something
that's up for discussions, I think that the current defaults for
atomicity are bad for users because they can and do lead to repository
corruption.
Patrick
[1]: https://thunk.org/tytso/blog/2009/03/15/dont-fear-the-fsync/
[2]: https://btrfs.wiki.kernel.org/index.php/FAQ (What are the crash guarantees of overwrite-by-rename)
[3]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/admin-guide/ext4.rst (see auto_da_alloc)
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v4 2/4] core.fsync: introduce granular fsync control
2022-02-14 7:04 ` Patrick Steinhardt
@ 2022-02-14 17:17 ` Junio C Hamano
2022-03-09 13:42 ` Patrick Steinhardt
0 siblings, 1 reply; 122+ messages in thread
From: Junio C Hamano @ 2022-02-14 17:17 UTC (permalink / raw)
To: Patrick Steinhardt
Cc: Neeraj Singh, Neeraj Singh via GitGitGadget, Git List,
Randall S. Becker, Bagas Sanjaya, Elijah Newren,
Ævar Arnfjörð Bjarmason, Neeraj K. Singh
Patrick Steinhardt <ps@pks.im> writes:
> To summarize my take: while the degree of durability may be something
> that's up for discussions, I think that the current defaults for
> atomicity are bad for users because they can and do lead to repository
> corruption.
Good summary.
If the user cares about fsynching loose object files in the right
way, we shouldn't leave loose ref files not following the safe
safety level, regardless of how this new core.fsync knobs would look
like.
I think we three are in agreement on that.
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v4 2/4] core.fsync: introduce granular fsync control
2022-02-14 17:17 ` Junio C Hamano
@ 2022-03-09 13:42 ` Patrick Steinhardt
2022-03-09 18:50 ` Ævar Arnfjörð Bjarmason
` (2 more replies)
0 siblings, 3 replies; 122+ messages in thread
From: Patrick Steinhardt @ 2022-03-09 13:42 UTC (permalink / raw)
To: Junio C Hamano
Cc: Neeraj Singh, Neeraj Singh via GitGitGadget, Git List,
Randall S. Becker, Bagas Sanjaya, Elijah Newren,
Ævar Arnfjörð Bjarmason, Neeraj K. Singh
[-- Attachment #1: Type: text/plain, Size: 1429 bytes --]
On Mon, Feb 14, 2022 at 09:17:31AM -0800, Junio C Hamano wrote:
> Patrick Steinhardt <ps@pks.im> writes:
>
> > To summarize my take: while the degree of durability may be something
> > that's up for discussions, I think that the current defaults for
> > atomicity are bad for users because they can and do lead to repository
> > corruption.
>
> Good summary.
>
> If the user cares about fsynching loose object files in the right
> way, we shouldn't leave loose ref files not following the safe
> safety level, regardless of how this new core.fsync knobs would look
> like.
>
> I think we three are in agreement on that.
Is there anything I can specifically do to help out with this topic? We
have again hit data loss in production because we don't sync loose refs
to disk before renaming them into place, so I'd really love to sort out
this issue somehow so that I can revive my patch series which fixes the
known repository corruption [1].
Alternatively, can we maybe find a way forward with applying a version
of my patch series without first settling the bigger question of how we
want the overall design to look like? In my opinion repository
corruption is a severe bug that needs to be fixed, and it doesn't feel
sensible to block such a fix over a discussion that potentially will
take a long time to settle.
Patrick
[1]: http://public-inbox.org/git/cover.1636544377.git.ps@pks.im/
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v4 2/4] core.fsync: introduce granular fsync control
2022-03-09 13:42 ` Patrick Steinhardt
@ 2022-03-09 18:50 ` Ævar Arnfjörð Bjarmason
2022-03-09 20:03 ` Junio C Hamano
2022-03-09 20:05 ` Neeraj Singh
2 siblings, 0 replies; 122+ messages in thread
From: Ævar Arnfjörð Bjarmason @ 2022-03-09 18:50 UTC (permalink / raw)
To: Patrick Steinhardt
Cc: Junio C Hamano, Neeraj Singh, Neeraj Singh via GitGitGadget,
Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren,
Neeraj K. Singh
On Wed, Mar 09 2022, Patrick Steinhardt wrote:
> [[PGP Signed Part:Undecided]]
> On Mon, Feb 14, 2022 at 09:17:31AM -0800, Junio C Hamano wrote:
>> Patrick Steinhardt <ps@pks.im> writes:
>>
>> > To summarize my take: while the degree of durability may be something
>> > that's up for discussions, I think that the current defaults for
>> > atomicity are bad for users because they can and do lead to repository
>> > corruption.
>>
>> Good summary.
>>
>> If the user cares about fsynching loose object files in the right
>> way, we shouldn't leave loose ref files not following the safe
>> safety level, regardless of how this new core.fsync knobs would look
>> like.
>>
>> I think we three are in agreement on that.
>
> Is there anything I can specifically do to help out with this topic? We
> have again hit data loss in production because we don't sync loose refs
> to disk before renaming them into place, so I'd really love to sort out
> this issue somehow so that I can revive my patch series which fixes the
> known repository corruption [1].
>
> Alternatively, can we maybe find a way forward with applying a version
> of my patch series without first settling the bigger question of how we
> want the overall design to look like? In my opinion repository
> corruption is a severe bug that needs to be fixed, and it doesn't feel
> sensible to block such a fix over a discussion that potentially will
> take a long time to settle.
>
> Patrick
>
> [1]: http://public-inbox.org/git/cover.1636544377.git.ps@pks.im/
I share that view. I was wondering how this topic fizzled out the other
day, but then promptly forgot about it.
I think the best thing at this point (hint hint!) would be for someone
in the know to (re-)submit the various patches appropriate to move this
forward. Whether that's just this series, part of it, or some/both of
those + patches from you and Eric and this point I don't know/remember.
But just to be explicitly clear, as probably the person most responsible
for pushing this towards the "bigger question of [...] overall
design".
I just wanted to facilitate a discussion that would result in the
various stakeholders who wanted to add some fsync-related config coming
up with something that's mutually compatible, and I think the design
from Neeraj in this series fits that purpose, is Good Enough etc.
I.e. the actually important and IMO blockers were all resolved, e.g. not
having an fsync configuration that older git versions would needlessly
die on, and not painting ourselves into a corner where
e.g. core.fsync=false or something was squatted on by something other
than a "no fsync, whatsoever" etc.
(But I haven't looked at it again just now, so...)
Anyway, just trying to be explicit that to whatever extent this was held
up by questions/comments of mine I'm very happy to see this go forward.
As you (basically) say we shouldn't lose sight of ongoing data loss in
this area because of some config bikeshedding :)
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v4 2/4] core.fsync: introduce granular fsync control
2022-03-09 13:42 ` Patrick Steinhardt
2022-03-09 18:50 ` Ævar Arnfjörð Bjarmason
@ 2022-03-09 20:03 ` Junio C Hamano
2022-03-10 12:33 ` Patrick Steinhardt
2022-03-09 20:05 ` Neeraj Singh
2 siblings, 1 reply; 122+ messages in thread
From: Junio C Hamano @ 2022-03-09 20:03 UTC (permalink / raw)
To: Patrick Steinhardt
Cc: Neeraj Singh, Neeraj Singh via GitGitGadget, Git List,
Randall S. Becker, Bagas Sanjaya, Elijah Newren,
Ævar Arnfjörð Bjarmason, Neeraj K. Singh
Patrick Steinhardt <ps@pks.im> writes:
>> If the user cares about fsynching loose object files in the right
>> way, we shouldn't leave loose ref files not following the safe
>> safety level, regardless of how this new core.fsync knobs would look
>> like.
>>
>> I think we three are in agreement on that.
>
> Is there anything I can specifically do to help out with this topic? We
> have again hit data loss in production because we don't sync loose refs
> to disk before renaming them into place, so I'd really love to sort out
> this issue somehow so that I can revive my patch series which fixes the
> known repository corruption [1].
How about doing a series to unconditionally sync loose ref creation
and modification?
Alternatively, we could link it to the existing configuration to
control synching of object files.
I do not think core.fsyncObjectFiles having "object" in its name is
a good reason not to think those who set it to true only care about
the loose object files and nothing else. It is more sensible to
consider that those who set it to true cares about the repository
integrity more than those who set it to false, I would think.
But that (i.e. doing it conditionally and choose which knob to use)
is one extra thing that needs justification, so starting from
unconditional fsync_or_die() may be the best way to ease it in.
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v4 2/4] core.fsync: introduce granular fsync control
2022-03-09 13:42 ` Patrick Steinhardt
2022-03-09 18:50 ` Ævar Arnfjörð Bjarmason
2022-03-09 20:03 ` Junio C Hamano
@ 2022-03-09 20:05 ` Neeraj Singh
2 siblings, 0 replies; 122+ messages in thread
From: Neeraj Singh @ 2022-03-09 20:05 UTC (permalink / raw)
To: Patrick Steinhardt
Cc: Junio C Hamano, Neeraj Singh via GitGitGadget, Git List,
Randall S. Becker, Bagas Sanjaya, Elijah Newren,
Ævar Arnfjörð Bjarmason, Neeraj K. Singh
On Wed, Mar 9, 2022 at 5:42 AM Patrick Steinhardt <ps@pks.im> wrote:
>
> On Mon, Feb 14, 2022 at 09:17:31AM -0800, Junio C Hamano wrote:
> > Patrick Steinhardt <ps@pks.im> writes:
> >
> > > To summarize my take: while the degree of durability may be something
> > > that's up for discussions, I think that the current defaults for
> > > atomicity are bad for users because they can and do lead to repository
> > > corruption.
> >
> > Good summary.
> >
> > If the user cares about fsynching loose object files in the right
> > way, we shouldn't leave loose ref files not following the safe
> > safety level, regardless of how this new core.fsync knobs would look
> > like.
> >
> > I think we three are in agreement on that.
>
> Is there anything I can specifically do to help out with this topic? We
> have again hit data loss in production because we don't sync loose refs
> to disk before renaming them into place, so I'd really love to sort out
> this issue somehow so that I can revive my patch series which fixes the
> known repository corruption [1].
>
> Alternatively, can we maybe find a way forward with applying a version
> of my patch series without first settling the bigger question of how we
> want the overall design to look like? In my opinion repository
> corruption is a severe bug that needs to be fixed, and it doesn't feel
> sensible to block such a fix over a discussion that potentially will
> take a long time to settle.
>
> Patrick
>
> [1]: http://public-inbox.org/git/cover.1636544377.git.ps@pks.im/
Hi Patrick,
Thanks for reviving this discussion. I've updated the PR on
GitGitGadget with a rebase
onto the current 'main' branch and some minor build fixes. I've also
revamped the aggregate options
and documentation to be more inline with Junio's suggestion of having
'levels of safety' that we steer
the user towards. I'm still keeping the detailed options, but
hopefully the guidance is clear enough to
avoid confusion.
I'd be happy to make any point fixes as necessary to get that branch
into proper shape for
upstream, if we've gotten to the point where we don't want to change
the fundamental design.
I agree with Patrick that the detailed knobs are primarily for use by
hosters like GitLab and GitHub.
Please expect a v5 today.
Thanks,
Neeraj
^ permalink raw reply [flat|nested] 122+ messages in thread
* [PATCH v5 0/5] A design for future-proofing fsync() configuration
2022-02-01 3:33 ` [PATCH v4 " Neeraj K. Singh via GitGitGadget
` (3 preceding siblings ...)
2022-02-01 3:33 ` [PATCH v4 4/4] core.fsync: add a `derived-metadata` aggregate option Neeraj Singh via GitGitGadget
@ 2022-03-09 23:03 ` Neeraj K. Singh via GitGitGadget
2022-03-09 23:03 ` [PATCH v5 1/5] wrapper: move inclusion of CSPRNG headers the wrapper.c file Neeraj Singh via GitGitGadget
` (9 more replies)
4 siblings, 10 replies; 122+ messages in thread
From: Neeraj K. Singh via GitGitGadget @ 2022-03-09 23:03 UTC (permalink / raw)
To: git
Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps, sandals,
Neeraj K. Singh
This is an implementation of an extensible configuration mechanism for
fsyncing persistent components of a repo.
The main goals are to separate the "what" to sync from the "how". There are
now two settings: core.fsync - Control the 'what', including the index.
core.fsyncMethod - Control the 'how'. Currently we support writeout-only and
full fsync.
Syncing of refs can be layered on top of core.fsync. And batch mode will be
layered on core.fsyncMethod. Once this series reaches 'seen', I'll submit
ns/batched-fsync to introduce batch mode. Please see
https://github.com/gitgitgadget/git/pull/1134.
core.fsyncObjectfiles is removed and we will issue a deprecation warning if
it's seen.
I'd like to get agreement on this direction before submitting batch mode to
the list. The batch mode series is available to view at
Please see [1], [2], and [3] for discussions that led to this series.
After this change, new persistent data files added to the repo will need to
be added to the fsync_component enum and documented in the
Documentation/config/core.txt text.
V5 changes:
* Rebase onto main at c2162907e9
* Add a patch to move CSPRNG platform includes to wrapper.c. This avoids
build errors in compat/win32/flush.c and other files.
* Move the documentation and aggregate options to the final patch in the
series.
* Define new aggregate options and guidance in line with Junio's suggestion
to present the user with 'levels of safety' rather than a morass of
detailed options.
V4 changes:
* Rebase onto master at b23dac905bd.
* Add a comment to write_pack_file indicating why we don't fsync when
writing to stdout.
* I kept the configuration schema as-is rather than switching to
multi-value. The thinking here is that a stateless last-one-wins config
schema (comma separated) will make it easier to achieve some holistic
self-consistent fsync configuration for a particular repo.
V3 changes:
* Remove relative path from git-compat-util.h include [4].
* Updated newly added warning texts to have more context for localization
[4].
* Fixed tab spacing in enum fsync_action
* Moved the fsync looping out to a helper and do it consistently. [4]
* Changed commit description to use camelCase for config names. [5]
* Add an optional fourth patch with derived-metadata so that the user can
exclude a forward-compatible set of things that should be recomputable
given existing data.
V2 changes:
* Updated the documentation for core.fsyncmethod to be less certain.
writeout-only probably does not do the right thing on Linux.
* Split out the core.fsync=index change into its own commit.
* Rename REPO_COMPONENT to FSYNC_COMPONENT. This is really specific to
fsyncing, so the name should reflect that.
* Re-add missing Makefile change for SYNC_FILE_RANGE.
* Tested writeout-only mode, index syncing, and general config settings.
[1] https://lore.kernel.org/git/211110.86r1bogg27.gmgdl@evledraar.gmail.com/
[2]
https://lore.kernel.org/git/dd65718814011eb93ccc4428f9882e0f025224a6.1636029491.git.ps@pks.im/
[3]
https://lore.kernel.org/git/pull.1076.git.git.1629856292.gitgitgadget@gmail.com/
[4]
https://lore.kernel.org/git/CANQDOdf8C4-haK9=Q_J4Cid8bQALnmGDm=SvatRbaVf+tkzqLw@mail.gmail.com/
[5] https://lore.kernel.org/git/211207.861r2opplg.gmgdl@evledraar.gmail.com/
Neeraj Singh (5):
wrapper: move inclusion of CSPRNG headers the wrapper.c file
core.fsyncmethod: add writeout-only mode
core.fsync: introduce granular fsync control
core.fsync: new option to harden the index
core.fsync: documentation and user-friendly aggregate options
Documentation/config/core.txt | 51 +++++++++++++---
Makefile | 6 ++
builtin/fast-import.c | 2 +-
builtin/index-pack.c | 4 +-
builtin/pack-objects.c | 24 +++++---
bulk-checkin.c | 5 +-
cache.h | 53 ++++++++++++++++-
commit-graph.c | 3 +-
compat/mingw.h | 3 +
compat/win32/flush.c | 28 +++++++++
compat/winansi.c | 5 --
config.c | 92 ++++++++++++++++++++++++++++-
config.mak.uname | 3 +
configure.ac | 8 +++
contrib/buildsystems/CMakeLists.txt | 16 +++--
csum-file.c | 5 +-
csum-file.h | 3 +-
environment.c | 3 +-
git-compat-util.h | 36 +++++++----
midx.c | 3 +-
object-file.c | 3 +-
pack-bitmap-write.c | 3 +-
pack-write.c | 13 ++--
read-cache.c | 19 ++++--
wrapper.c | 78 ++++++++++++++++++++++++
write-or-die.c | 11 ++--
26 files changed, 413 insertions(+), 67 deletions(-)
create mode 100644 compat/win32/flush.c
base-commit: c2162907e9aa884bdb70208389cb99b181620d51
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1093%2Fneerajsi-msft%2Fns%2Fcore-fsync-v5
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1093/neerajsi-msft/ns/core-fsync-v5
Pull-Request: https://github.com/gitgitgadget/git/pull/1093
Range-diff vs v4:
-: ----------- > 1: 685b1db8880 wrapper: move inclusion of CSPRNG headers the wrapper.c file
1: 51a218d100d ! 2: da8cfc10bb4 core.fsyncmethod: add writeout-only mode
@@ contrib/buildsystems/CMakeLists.txt
@@ contrib/buildsystems/CMakeLists.txt: if(CMAKE_SYSTEM_NAME STREQUAL "Windows")
NOGDI OBJECT_CREATION_MODE=1 __USE_MINGW_ANSI_STDIO=0
USE_NED_ALLOCATOR OVERRIDE_STRDUP MMAP_PREVENTS_DELETE USE_WIN32_MMAP
- UNICODE _UNICODE HAVE_WPGMPTR ENSURE_MSYSTEM_IS_SET)
+ UNICODE _UNICODE HAVE_WPGMPTR ENSURE_MSYSTEM_IS_SET HAVE_RTLGENRANDOM)
- list(APPEND compat_SOURCES compat/mingw.c compat/winansi.c compat/win32/path-utils.c
-+ list(APPEND compat_SOURCES compat/mingw.c compat/winansi.c
-+ compat/win32/flush.c compat/win32/path-utils.c
- compat/win32/pthread.c compat/win32mmap.c compat/win32/syslog.c
- compat/win32/trace2_win32_process_info.c compat/win32/dirent.c
- compat/nedmalloc/nedmalloc.c compat/strdup.c)
+- compat/win32/pthread.c compat/win32mmap.c compat/win32/syslog.c
+- compat/win32/trace2_win32_process_info.c compat/win32/dirent.c
+- compat/nedmalloc/nedmalloc.c compat/strdup.c)
++ list(APPEND compat_SOURCES
++ compat/mingw.c
++ compat/winansi.c
++ compat/win32/flush.c
++ compat/win32/path-utils.c
++ compat/win32/pthread.c
++ compat/win32mmap.c
++ compat/win32/syslog.c
++ compat/win32/trace2_win32_process_info.c
++ compat/win32/dirent.c
++ compat/nedmalloc/nedmalloc.c
++ compat/strdup.c)
+ set(NO_UNIX_SOCKETS 1)
+
+ elseif(CMAKE_SYSTEM_NAME STREQUAL "Linux")
## environment.c ##
@@ environment.c: int zlib_compression_level = Z_BEST_SPEED;
@@ wrapper.c: int xmkstemp_mode(char *filename_template, int mode)
+
+#ifdef __APPLE__
+ /*
-+ * on macOS, fsync just causes filesystem cache writeback but does not
-+ * flush hardware caches.
++ * On macOS, fsync just causes filesystem cache writeback but
++ * does not flush hardware caches.
+ */
+ return fsync_loop(fd);
+#endif
+
+#ifdef HAVE_SYNC_FILE_RANGE
+ /*
-+ * On linux 2.6.17 and above, sync_file_range is the way to issue
-+ * a writeback without a hardware flush. An offset of 0 and size of 0
-+ * indicates writeout of the entire file and the wait flags ensure that all
-+ * dirty data is written to the disk (potentially in a disk-side cache)
-+ * before we continue.
++ * On linux 2.6.17 and above, sync_file_range is the way to
++ * issue a writeback without a hardware flush. An offset of
++ * 0 and size of 0 indicates writeout of the entire file and the
++ * wait flags ensure that all dirty data is written to the disk
++ * (potentially in a disk-side cache) before we continue.
+ */
+
+ return sync_file_range(fd, 0, 0, SYNC_FILE_RANGE_WAIT_BEFORE |
@@ wrapper.c: int xmkstemp_mode(char *filename_template, int mode)
+
+ case FSYNC_HARDWARE_FLUSH:
+ /*
-+ * On some platforms fsync may return EINTR. Try again in this
-+ * case, since callers asking for a hardware flush may die if
-+ * this function returns an error.
++ * On macOS, a special fcntl is required to really flush the
++ * caches within the storage controller. As of this writing,
++ * this is a very expensive operation on Apple SSDs.
+ */
+#ifdef __APPLE__
+ return fcntl(fd, F_FULLFSYNC);
2: 7a164ba9571 ! 3: e31886717b4 core.fsync: introduce granular fsync control
@@ Commit message
syncable components:
* We issue a warning rather than an error for unrecognized
components, so new configs can be used with old Git versions.
- * We support negation, so users can choose one of the default
- aggregate options and then remove components that they don't
- want. The user would then harden any new components added in
- a Git version update.
+ * We support negation, so users can choose one of the aggregate
+ options and then remove components that they don't want.
+ Aggregate options are defined in a later patch in this series.
This also supports the common request of doing absolutely no
fysncing with the `core.fsync=none` value, which is expected
to make the test suite faster.
+ Complete documentation for the new setting is included in a later patch
+ in the series so that it can be reviewed in final form.
+
Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
## Documentation/config/core.txt ##
-@@ Documentation/config/core.txt: core.whitespace::
- is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent`
- errors. The default tab width is 8. Allowed values are 1 to 63.
-
-+core.fsync::
-+ A comma-separated list of parts of the repository which should be
-+ hardened via the core.fsyncMethod when created or modified. You can
-+ disable hardening of any component by prefixing it with a '-'. Later
-+ items take precedence over earlier ones in the list. For example,
-+ `core.fsync=all,-pack-metadata` means "harden everything except pack
-+ metadata." Items that are not hardened may be lost in the event of an
-+ unclean system shutdown.
-++
-+* `none` disables fsync completely. This must be specified alone.
-+* `loose-object` hardens objects added to the repo in loose-object form.
-+* `pack` hardens objects added to the repo in packfile form.
-+* `pack-metadata` hardens packfile bitmaps and indexes.
-+* `commit-graph` hardens the commit graph file.
-+* `objects` is an aggregate option that includes `loose-objects`, `pack`,
-+ `pack-metadata`, and `commit-graph`.
-+* `default` is an aggregate option that is equivalent to `objects,-loose-object`
-+* `all` is an aggregate option that syncs all individual components above.
-+
- core.fsyncMethod::
- A value indicating the strategy Git will use to harden repository data
- using fsync and related primitives.
@@ Documentation/config/core.txt: core.fsyncMethod::
filesystem and storage hardware, data added to the repository may not be
durable in the event of a system crash. This is the default mode on macOS.
@@ cache.h: void reset_shared_repository(void);
+ FSYNC_COMPONENT_PACK_METADATA | \
+ FSYNC_COMPONENT_COMMIT_GRAPH)
+
-+#define FSYNC_COMPONENTS_OBJECTS (FSYNC_COMPONENT_LOOSE_OBJECT | \
-+ FSYNC_COMPONENT_PACK | \
-+ FSYNC_COMPONENT_PACK_METADATA | \
-+ FSYNC_COMPONENT_COMMIT_GRAPH)
-+
-+#define FSYNC_COMPONENTS_ALL (FSYNC_COMPONENT_LOOSE_OBJECT | \
-+ FSYNC_COMPONENT_PACK | \
-+ FSYNC_COMPONENT_PACK_METADATA | \
-+ FSYNC_COMPONENT_COMMIT_GRAPH)
-+
-+
+/*
+ * A bitmask indicating which components of the repo should be fsynced.
+ */
@@ cache.h: int copy_file_with_time(const char *dst, const char *src, int mode);
void write_or_die(int fd, const void *buf, size_t count);
void fsync_or_die(int fd, const char *);
-+inline void fsync_component_or_die(enum fsync_component component, int fd, const char *msg)
++static inline void fsync_component_or_die(enum fsync_component component, int fd, const char *msg)
+{
+ if (fsync_components & component)
+ fsync_or_die(fd, msg);
@@ config.c: static int git_parse_maybe_bool_text(const char *value)
+ { "pack", FSYNC_COMPONENT_PACK },
+ { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA },
+ { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH },
-+ { "objects", FSYNC_COMPONENTS_OBJECTS },
-+ { "default", FSYNC_COMPONENTS_DEFAULT },
-+ { "all", FSYNC_COMPONENTS_ALL },
+};
+
+static enum fsync_component parse_fsync_components(const char *var, const char *string)
@@ config.c: static int git_parse_maybe_bool_text(const char *value)
+ enum fsync_component output = 0;
+
+ if (!strcmp(string, "none"))
-+ return output;
++ return FSYNC_COMPONENT_NONE;
+
+ while (string) {
+ int i;
@@ midx.c: static int write_midx_internal(const char *object_dir,
+ CSUM_FSYNC | CSUM_HASH_IN_STREAM);
free_chunkfile(cf);
- if (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP))
+ if (flags & MIDX_WRITE_REV_INDEX &&
## object-file.c ##
@@ object-file.c: int hash_object_file(const struct git_hash_algo *algo, const void *buf,
3: f217dba77a1 ! 4: 9da808ba743 core.fsync: new option to harden the index
@@ Commit message
Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
- ## Documentation/config/core.txt ##
-@@ Documentation/config/core.txt: core.fsync::
- * `pack` hardens objects added to the repo in packfile form.
- * `pack-metadata` hardens packfile bitmaps and indexes.
- * `commit-graph` hardens the commit graph file.
-+* `index` hardens the index when it is modified.
- * `objects` is an aggregate option that includes `loose-objects`, `pack`,
- `pack-metadata`, and `commit-graph`.
- * `default` is an aggregate option that is equivalent to `objects,-loose-object`
-
## cache.h ##
@@ cache.h: enum fsync_component {
FSYNC_COMPONENT_PACK = 1 << 1,
@@ cache.h: enum fsync_component {
};
#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \
-@@ cache.h: enum fsync_component {
- #define FSYNC_COMPONENTS_ALL (FSYNC_COMPONENT_LOOSE_OBJECT | \
- FSYNC_COMPONENT_PACK | \
- FSYNC_COMPONENT_PACK_METADATA | \
-- FSYNC_COMPONENT_COMMIT_GRAPH)
-+ FSYNC_COMPONENT_COMMIT_GRAPH | \
-+ FSYNC_COMPONENT_INDEX)
-
-
- /*
## config.c ##
@@ config.c: static const struct fsync_component_entry {
@@ config.c: static const struct fsync_component_entry {
{ "pack-metadata", FSYNC_COMPONENT_PACK_METADATA },
{ "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH },
+ { "index", FSYNC_COMPONENT_INDEX },
- { "objects", FSYNC_COMPONENTS_OBJECTS },
- { "default", FSYNC_COMPONENTS_DEFAULT },
- { "all", FSYNC_COMPONENTS_ALL },
+ };
+
+ static enum fsync_component parse_fsync_components(const char *var, const char *string)
## read-cache.c ##
@@ read-cache.c: static int record_ieot(void)
4: 5c22a41c1f3 ! 5: 2d71346b10e core.fsync: add a `derived-metadata` aggregate option
@@ Metadata
Author: Neeraj Singh <neerajsi@microsoft.com>
## Commit message ##
- core.fsync: add a `derived-metadata` aggregate option
+ core.fsync: documentation and user-friendly aggregate options
- This commit adds an aggregate option that currently includes the
- commit-graph file and pack metadata (indexes and bitmaps).
+ This commit adds aggregate options for the core.fsync setting that are
+ more user-friendly. These options are specified in terms of 'levels of
+ safety', indicating which Git operations are considered to be sync
+ points for durability.
- The user may want to exclude this set from durability since they can be
- recomputed from other data if they wind up corrupt or missing.
-
- This is split out from the other patches in the series since it is
- an optional nice-to-have that might be controversial.
+ The new documentation is also included here in its entirety for ease of
+ review.
Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
## Documentation/config/core.txt ##
-@@ Documentation/config/core.txt: core.fsync::
- * `pack-metadata` hardens packfile bitmaps and indexes.
- * `commit-graph` hardens the commit graph file.
- * `index` hardens the index when it is modified.
--* `objects` is an aggregate option that includes `loose-objects`, `pack`,
-- `pack-metadata`, and `commit-graph`.
--* `default` is an aggregate option that is equivalent to `objects,-loose-object`
-+* `objects` is an aggregate option that includes `loose-objects` and `pack`.
-+* `derived-metadata` is an aggregate option that includes `pack-metadata` and `commit-graph`.
-+* `default` is an aggregate option that is equivalent to `objects,derived-metadata,-loose-object`
- * `all` is an aggregate option that syncs all individual components above.
+@@ Documentation/config/core.txt: core.whitespace::
+ is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent`
+ errors. The default tab width is 8. Allowed values are 1 to 63.
++core.fsync::
++ A comma-separated list of parts of the repository which should be
++ hardened via the core.fsyncMethod when created or modified. You can
++ disable hardening of any component by prefixing it with a '-'. Later
++ items take precedence over earlier ones in the comma-separated list.
++ For example, `core.fsync=all,-pack-metadata` means "harden everything
++ except pack metadata." Items that are not hardened may be lost in the
++ event of an unclean system shutdown. Unless you have special
++ requirements, it is recommended that you leave this option as default
++ or pick one of `committed`, `added`, or `all`.
+++
++* `none` disables fsync completely. This value must be specified alone.
++* `loose-object` hardens objects added to the repo in loose-object form.
++* `pack` hardens objects added to the repo in packfile form.
++* `pack-metadata` hardens packfile bitmaps and indexes.
++* `commit-graph` hardens the commit graph file.
++* `index` hardens the index when it is modified.
++* `objects` is an aggregate option that is equivalent to
++ `loose-object,pack`.
++* `derived-metadata` is an aggregate option that is equivalent to
++ `pack-metadata,commit-graph`.
++* `default` is an aggregate option that is equivalent to
++ `objects,derived-metadata,-loose-object`. This mode is enabled by default.
++ It has good performance, but risks losing recent work if the system shuts
++ down uncleanly, since commits, trees, and blobs in loose-object form may be
++ lost.
++* `committed` is an aggregate option that is currently equivalent to
++ `objects`. This mode sacrifices some performance to ensure that all work
++ that is committed to the repository with `git commit` or similar commands
++ is preserved.
++* `added` is an aggregate option that is currently equivalent to
++ `committed,index`. This mode sacrifices additional performance to
++ ensure that the results of commands like `git add` and similar operations
++ are preserved.
++* `all` is an aggregate option that syncs all individual components above.
++
core.fsyncMethod::
+ A value indicating the strategy Git will use to harden repository data
+ using fsync and related primitives.
## cache.h ##
@@ cache.h: enum fsync_component {
- FSYNC_COMPONENT_COMMIT_GRAPH)
+ FSYNC_COMPONENT_INDEX = 1 << 4,
+ };
- #define FSYNC_COMPONENTS_OBJECTS (FSYNC_COMPONENT_LOOSE_OBJECT | \
-- FSYNC_COMPONENT_PACK | \
+-#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \
- FSYNC_COMPONENT_PACK_METADATA | \
- FSYNC_COMPONENT_COMMIT_GRAPH)
++#define FSYNC_COMPONENTS_OBJECTS (FSYNC_COMPONENT_LOOSE_OBJECT | \
+ FSYNC_COMPONENT_PACK)
+
+#define FSYNC_COMPONENTS_DERIVED_METADATA (FSYNC_COMPONENT_PACK_METADATA | \
+ FSYNC_COMPONENT_COMMIT_GRAPH)
++
++#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENTS_OBJECTS | \
++ FSYNC_COMPONENTS_DERIVED_METADATA | \
++ ~FSYNC_COMPONENT_LOOSE_OBJECT)
++
++#define FSYNC_COMPONENTS_COMMITTED (FSYNC_COMPONENTS_OBJECTS)
++
++#define FSYNC_COMPONENTS_ADDED (FSYNC_COMPONENTS_COMMITTED | \
++ FSYNC_COMPONENT_INDEX)
++
++#define FSYNC_COMPONENTS_ALL (FSYNC_COMPONENT_LOOSE_OBJECT | \
++ FSYNC_COMPONENT_PACK | \
++ FSYNC_COMPONENT_PACK_METADATA | \
++ FSYNC_COMPONENT_COMMIT_GRAPH | \
++ FSYNC_COMPONENT_INDEX)
- #define FSYNC_COMPONENTS_ALL (FSYNC_COMPONENT_LOOSE_OBJECT | \
- FSYNC_COMPONENT_PACK | \
+ /*
+ * A bitmask indicating which components of the repo should be fsynced.
## config.c ##
@@ config.c: static const struct fsync_component_entry {
+ { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA },
{ "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH },
{ "index", FSYNC_COMPONENT_INDEX },
- { "objects", FSYNC_COMPONENTS_OBJECTS },
++ { "objects", FSYNC_COMPONENTS_OBJECTS },
+ { "derived-metadata", FSYNC_COMPONENTS_DERIVED_METADATA },
- { "default", FSYNC_COMPONENTS_DEFAULT },
- { "all", FSYNC_COMPONENTS_ALL },
++ { "default", FSYNC_COMPONENTS_DEFAULT },
++ { "committed", FSYNC_COMPONENTS_COMMITTED },
++ { "added", FSYNC_COMPONENTS_ADDED },
++ { "all", FSYNC_COMPONENTS_ALL },
};
+
+ static enum fsync_component parse_fsync_components(const char *var, const char *string)
--
gitgitgadget
^ permalink raw reply [flat|nested] 122+ messages in thread
* [PATCH v5 1/5] wrapper: move inclusion of CSPRNG headers the wrapper.c file
2022-03-09 23:03 ` [PATCH v5 0/5] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget
@ 2022-03-09 23:03 ` Neeraj Singh via GitGitGadget
2022-03-09 23:29 ` Junio C Hamano
2022-03-10 1:26 ` brian m. carlson
2022-03-09 23:03 ` [PATCH v5 2/5] core.fsyncmethod: add writeout-only mode Neeraj Singh via GitGitGadget
` (8 subsequent siblings)
9 siblings, 2 replies; 122+ messages in thread
From: Neeraj Singh via GitGitGadget @ 2022-03-09 23:03 UTC (permalink / raw)
To: git
Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps, sandals,
Neeraj K. Singh, Neeraj Singh
From: Neeraj Singh <neerajsi@microsoft.com>
Including NTSecAPI.h in git-compat-util.h causes build errors in any
other file that includes winternl.h. That file was included in order to
get access to the RtlGenRandom cryptographically secure PRNG. This
change scopes the inclusion of all PRNG headers to just the wrapper.c
file, which is the only place it is really needed.
Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
---
compat/winansi.c | 5 -----
git-compat-util.h | 12 ------------
wrapper.c | 14 ++++++++++++++
3 files changed, 14 insertions(+), 17 deletions(-)
diff --git a/compat/winansi.c b/compat/winansi.c
index 936a80a5f00..3abe8dd5a27 100644
--- a/compat/winansi.c
+++ b/compat/winansi.c
@@ -4,11 +4,6 @@
#undef NOGDI
-/*
- * Including the appropriate header file for RtlGenRandom causes MSVC to see a
- * redefinition of types in an incompatible way when including headers below.
- */
-#undef HAVE_RTLGENRANDOM
#include "../git-compat-util.h"
#include <wingdi.h>
#include <winreg.h>
diff --git a/git-compat-util.h b/git-compat-util.h
index 876907b9df4..a25ebb822ee 100644
--- a/git-compat-util.h
+++ b/git-compat-util.h
@@ -197,12 +197,6 @@
#endif
#include <windows.h>
#define GIT_WINDOWS_NATIVE
-#ifdef HAVE_RTLGENRANDOM
-/* This is required to get access to RtlGenRandom. */
-#define SystemFunction036 NTAPI SystemFunction036
-#include <NTSecAPI.h>
-#undef SystemFunction036
-#endif
#endif
#include <unistd.h>
@@ -273,12 +267,6 @@
#else
#include <stdint.h>
#endif
-#ifdef HAVE_ARC4RANDOM_LIBBSD
-#include <bsd/stdlib.h>
-#endif
-#ifdef HAVE_GETRANDOM
-#include <sys/random.h>
-#endif
#ifdef NO_INTPTR_T
/*
* On I16LP32, ILP32 and LP64 "long" is the safe bet, however
diff --git a/wrapper.c b/wrapper.c
index 3258cdb171f..2a1aade473b 100644
--- a/wrapper.c
+++ b/wrapper.c
@@ -4,6 +4,20 @@
#include "cache.h"
#include "config.h"
+#ifdef HAVE_RTLGENRANDOM
+/* This is required to get access to RtlGenRandom. */
+#define SystemFunction036 NTAPI SystemFunction036
+#include <NTSecAPI.h>
+#undef SystemFunction036
+#endif
+
+#ifdef HAVE_ARC4RANDOM_LIBBSD
+#include <bsd/stdlib.h>
+#endif
+#ifdef HAVE_GETRANDOM
+#include <sys/random.h>
+#endif
+
static int memory_limit_check(size_t size, int gentle)
{
static size_t limit = 0;
--
gitgitgadget
^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v5 2/5] core.fsyncmethod: add writeout-only mode
2022-03-09 23:03 ` [PATCH v5 0/5] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget
2022-03-09 23:03 ` [PATCH v5 1/5] wrapper: move inclusion of CSPRNG headers the wrapper.c file Neeraj Singh via GitGitGadget
@ 2022-03-09 23:03 ` Neeraj Singh via GitGitGadget
2022-03-09 23:48 ` Junio C Hamano
2022-03-09 23:03 ` [PATCH v5 3/5] core.fsync: introduce granular fsync control Neeraj Singh via GitGitGadget
` (7 subsequent siblings)
9 siblings, 1 reply; 122+ messages in thread
From: Neeraj Singh via GitGitGadget @ 2022-03-09 23:03 UTC (permalink / raw)
To: git
Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps, sandals,
Neeraj K. Singh, Neeraj Singh
From: Neeraj Singh <neerajsi@microsoft.com>
This commit introduces the `core.fsyncMethod` configuration
knob, which can currently be set to `fsync` or `writeout-only`.
The new writeout-only mode attempts to tell the operating system to
flush its in-memory page cache to the storage hardware without issuing a
CACHE_FLUSH command to the storage controller.
Writeout-only fsync is significantly faster than a vanilla fsync on
common hardware, since data is written to a disk-side cache rather than
all the way to a durable medium. Later changes in this patch series will
take advantage of this primitive to implement batching of hardware
flushes.
When git_fsync is called with FSYNC_WRITEOUT_ONLY, it may fail and the
caller is expected to do an ordinary fsync as needed.
On Apple platforms, the fsync system call does not issue a CACHE_FLUSH
directive to the storage controller. This change updates fsync to do
fcntl(F_FULLFSYNC) to make fsync actually durable. We maintain parity
with existing behavior on Apple platforms by setting the default value
of the new core.fsyncMethod option.
Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
---
Documentation/config/core.txt | 9 ++++
Makefile | 6 +++
cache.h | 7 ++++
compat/mingw.h | 3 ++
compat/win32/flush.c | 28 +++++++++++++
config.c | 12 ++++++
config.mak.uname | 3 ++
configure.ac | 8 ++++
contrib/buildsystems/CMakeLists.txt | 16 ++++++--
environment.c | 1 +
git-compat-util.h | 24 +++++++++++
wrapper.c | 64 +++++++++++++++++++++++++++++
write-or-die.c | 11 +++--
13 files changed, 184 insertions(+), 8 deletions(-)
create mode 100644 compat/win32/flush.c
diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
index c04f62a54a1..dbb134f7136 100644
--- a/Documentation/config/core.txt
+++ b/Documentation/config/core.txt
@@ -547,6 +547,15 @@ core.whitespace::
is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent`
errors. The default tab width is 8. Allowed values are 1 to 63.
+core.fsyncMethod::
+ A value indicating the strategy Git will use to harden repository data
+ using fsync and related primitives.
++
+* `fsync` uses the fsync() system call or platform equivalents.
+* `writeout-only` issues pagecache writeback requests, but depending on the
+ filesystem and storage hardware, data added to the repository may not be
+ durable in the event of a system crash. This is the default mode on macOS.
+
core.fsyncObjectFiles::
This boolean will enable 'fsync()' when writing object files.
+
diff --git a/Makefile b/Makefile
index 6f0b4b775fe..17fd9b023a4 100644
--- a/Makefile
+++ b/Makefile
@@ -411,6 +411,8 @@ all::
#
# Define HAVE_CLOCK_MONOTONIC if your platform has CLOCK_MONOTONIC.
#
+# Define HAVE_SYNC_FILE_RANGE if your platform has sync_file_range.
+#
# Define NEEDS_LIBRT if your platform requires linking with librt (glibc version
# before 2.17) for clock_gettime and CLOCK_MONOTONIC.
#
@@ -1897,6 +1899,10 @@ ifdef HAVE_CLOCK_MONOTONIC
BASIC_CFLAGS += -DHAVE_CLOCK_MONOTONIC
endif
+ifdef HAVE_SYNC_FILE_RANGE
+ BASIC_CFLAGS += -DHAVE_SYNC_FILE_RANGE
+endif
+
ifdef NEEDS_LIBRT
EXTLIBS += -lrt
endif
diff --git a/cache.h b/cache.h
index 04d4d2db25c..82f0194a3dd 100644
--- a/cache.h
+++ b/cache.h
@@ -995,6 +995,13 @@ extern char *git_replace_ref_base;
extern int fsync_object_files;
extern int use_fsync;
+
+enum fsync_method {
+ FSYNC_METHOD_FSYNC,
+ FSYNC_METHOD_WRITEOUT_ONLY
+};
+
+extern enum fsync_method fsync_method;
extern int core_preload_index;
extern int precomposed_unicode;
extern int protect_hfs;
diff --git a/compat/mingw.h b/compat/mingw.h
index c9a52ad64a6..6074a3d3ced 100644
--- a/compat/mingw.h
+++ b/compat/mingw.h
@@ -329,6 +329,9 @@ int mingw_getpagesize(void);
#define getpagesize mingw_getpagesize
#endif
+int win32_fsync_no_flush(int fd);
+#define fsync_no_flush win32_fsync_no_flush
+
struct rlimit {
unsigned int rlim_cur;
};
diff --git a/compat/win32/flush.c b/compat/win32/flush.c
new file mode 100644
index 00000000000..291f90ea940
--- /dev/null
+++ b/compat/win32/flush.c
@@ -0,0 +1,28 @@
+#include "git-compat-util.h"
+#include <winternl.h>
+#include "lazyload.h"
+
+int win32_fsync_no_flush(int fd)
+{
+ IO_STATUS_BLOCK io_status;
+
+#define FLUSH_FLAGS_FILE_DATA_ONLY 1
+
+ DECLARE_PROC_ADDR(ntdll.dll, NTSTATUS, NTAPI, NtFlushBuffersFileEx,
+ HANDLE FileHandle, ULONG Flags, PVOID Parameters, ULONG ParameterSize,
+ PIO_STATUS_BLOCK IoStatusBlock);
+
+ if (!INIT_PROC_ADDR(NtFlushBuffersFileEx)) {
+ errno = ENOSYS;
+ return -1;
+ }
+
+ memset(&io_status, 0, sizeof(io_status));
+ if (NtFlushBuffersFileEx((HANDLE)_get_osfhandle(fd), FLUSH_FLAGS_FILE_DATA_ONLY,
+ NULL, 0, &io_status)) {
+ errno = EINVAL;
+ return -1;
+ }
+
+ return 0;
+}
diff --git a/config.c b/config.c
index 383b1a4885b..f3ff80b01c9 100644
--- a/config.c
+++ b/config.c
@@ -1600,6 +1600,18 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
return 0;
}
+ if (!strcmp(var, "core.fsyncmethod")) {
+ if (!value)
+ return config_error_nonbool(var);
+ if (!strcmp(value, "fsync"))
+ fsync_method = FSYNC_METHOD_FSYNC;
+ else if (!strcmp(value, "writeout-only"))
+ fsync_method = FSYNC_METHOD_WRITEOUT_ONLY;
+ else
+ warning(_("ignoring unknown core.fsyncMethod value '%s'"), value);
+
+ }
+
if (!strcmp(var, "core.fsyncobjectfiles")) {
fsync_object_files = git_config_bool(var, value);
return 0;
diff --git a/config.mak.uname b/config.mak.uname
index 4352ea39e9b..404fff5dd04 100644
--- a/config.mak.uname
+++ b/config.mak.uname
@@ -57,6 +57,7 @@ ifeq ($(uname_S),Linux)
HAVE_CLOCK_MONOTONIC = YesPlease
# -lrt is needed for clock_gettime on glibc <= 2.16
NEEDS_LIBRT = YesPlease
+ HAVE_SYNC_FILE_RANGE = YesPlease
HAVE_GETDELIM = YesPlease
FREAD_READS_DIRECTORIES = UnfortunatelyYes
BASIC_CFLAGS += -DHAVE_SYSINFO
@@ -463,6 +464,7 @@ endif
CFLAGS =
BASIC_CFLAGS = -nologo -I. -Icompat/vcbuild/include -DWIN32 -D_CONSOLE -DHAVE_STRING_H -D_CRT_SECURE_NO_WARNINGS -D_CRT_NONSTDC_NO_DEPRECATE
COMPAT_OBJS = compat/msvc.o compat/winansi.o \
+ compat/win32/flush.o \
compat/win32/path-utils.o \
compat/win32/pthread.o compat/win32/syslog.o \
compat/win32/trace2_win32_process_info.o \
@@ -640,6 +642,7 @@ ifeq ($(uname_S),MINGW)
COMPAT_CFLAGS += -DSTRIP_EXTENSION=\".exe\"
COMPAT_OBJS += compat/mingw.o compat/winansi.o \
compat/win32/trace2_win32_process_info.o \
+ compat/win32/flush.o \
compat/win32/path-utils.o \
compat/win32/pthread.o compat/win32/syslog.o \
compat/win32/dirent.o
diff --git a/configure.ac b/configure.ac
index 5ee25ec95c8..6bd6bef1c44 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1082,6 +1082,14 @@ AC_COMPILE_IFELSE([CLOCK_MONOTONIC_SRC],
[AC_MSG_RESULT([no])
HAVE_CLOCK_MONOTONIC=])
GIT_CONF_SUBST([HAVE_CLOCK_MONOTONIC])
+
+#
+# Define HAVE_SYNC_FILE_RANGE=YesPlease if sync_file_range is available.
+GIT_CHECK_FUNC(sync_file_range,
+ [HAVE_SYNC_FILE_RANGE=YesPlease],
+ [HAVE_SYNC_FILE_RANGE])
+GIT_CONF_SUBST([HAVE_SYNC_FILE_RANGE])
+
#
# Define NO_SETITIMER if you don't have setitimer.
GIT_CHECK_FUNC(setitimer,
diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt
index e44232f85d3..3a9e6241660 100644
--- a/contrib/buildsystems/CMakeLists.txt
+++ b/contrib/buildsystems/CMakeLists.txt
@@ -261,10 +261,18 @@ if(CMAKE_SYSTEM_NAME STREQUAL "Windows")
NOGDI OBJECT_CREATION_MODE=1 __USE_MINGW_ANSI_STDIO=0
USE_NED_ALLOCATOR OVERRIDE_STRDUP MMAP_PREVENTS_DELETE USE_WIN32_MMAP
UNICODE _UNICODE HAVE_WPGMPTR ENSURE_MSYSTEM_IS_SET HAVE_RTLGENRANDOM)
- list(APPEND compat_SOURCES compat/mingw.c compat/winansi.c compat/win32/path-utils.c
- compat/win32/pthread.c compat/win32mmap.c compat/win32/syslog.c
- compat/win32/trace2_win32_process_info.c compat/win32/dirent.c
- compat/nedmalloc/nedmalloc.c compat/strdup.c)
+ list(APPEND compat_SOURCES
+ compat/mingw.c
+ compat/winansi.c
+ compat/win32/flush.c
+ compat/win32/path-utils.c
+ compat/win32/pthread.c
+ compat/win32mmap.c
+ compat/win32/syslog.c
+ compat/win32/trace2_win32_process_info.c
+ compat/win32/dirent.c
+ compat/nedmalloc/nedmalloc.c
+ compat/strdup.c)
set(NO_UNIX_SOCKETS 1)
elseif(CMAKE_SYSTEM_NAME STREQUAL "Linux")
diff --git a/environment.c b/environment.c
index fd0501e77a5..3e3620d759f 100644
--- a/environment.c
+++ b/environment.c
@@ -44,6 +44,7 @@ int zlib_compression_level = Z_BEST_SPEED;
int pack_compression_level = Z_DEFAULT_COMPRESSION;
int fsync_object_files;
int use_fsync = -1;
+enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT;
size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE;
size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT;
size_t delta_base_cache_limit = 96 * 1024 * 1024;
diff --git a/git-compat-util.h b/git-compat-util.h
index a25ebb822ee..41b97027457 100644
--- a/git-compat-util.h
+++ b/git-compat-util.h
@@ -1265,6 +1265,30 @@ __attribute__((format (printf, 1, 2))) NORETURN
void BUG(const char *fmt, ...);
#endif
+#ifdef __APPLE__
+#define FSYNC_METHOD_DEFAULT FSYNC_METHOD_WRITEOUT_ONLY
+#else
+#define FSYNC_METHOD_DEFAULT FSYNC_METHOD_FSYNC
+#endif
+
+enum fsync_action {
+ FSYNC_WRITEOUT_ONLY,
+ FSYNC_HARDWARE_FLUSH
+};
+
+/*
+ * Issues an fsync against the specified file according to the specified mode.
+ *
+ * FSYNC_WRITEOUT_ONLY attempts to use interfaces available on some operating
+ * systems to flush the OS cache without issuing a flush command to the storage
+ * controller. If those interfaces are unavailable, the function fails with
+ * ENOSYS.
+ *
+ * FSYNC_HARDWARE_FLUSH does an OS writeout and hardware flush to ensure that
+ * changes are durable. It is not expected to fail.
+ */
+int git_fsync(int fd, enum fsync_action action);
+
/*
* Preserves errno, prints a message, but gives no warning for ENOENT.
* Returns 0 on success, which includes trying to unlink an object that does
diff --git a/wrapper.c b/wrapper.c
index 2a1aade473b..2c44f71da42 100644
--- a/wrapper.c
+++ b/wrapper.c
@@ -553,6 +553,70 @@ int xmkstemp_mode(char *filename_template, int mode)
return fd;
}
+/*
+ * Some platforms return EINTR from fsync. Since fsync is invoked in some
+ * cases by a wrapper that dies on failure, do not expose EINTR to callers.
+ */
+static int fsync_loop(int fd)
+{
+ int err;
+
+ do {
+ err = fsync(fd);
+ } while (err < 0 && errno == EINTR);
+ return err;
+}
+
+int git_fsync(int fd, enum fsync_action action)
+{
+ switch (action) {
+ case FSYNC_WRITEOUT_ONLY:
+
+#ifdef __APPLE__
+ /*
+ * On macOS, fsync just causes filesystem cache writeback but
+ * does not flush hardware caches.
+ */
+ return fsync_loop(fd);
+#endif
+
+#ifdef HAVE_SYNC_FILE_RANGE
+ /*
+ * On linux 2.6.17 and above, sync_file_range is the way to
+ * issue a writeback without a hardware flush. An offset of
+ * 0 and size of 0 indicates writeout of the entire file and the
+ * wait flags ensure that all dirty data is written to the disk
+ * (potentially in a disk-side cache) before we continue.
+ */
+
+ return sync_file_range(fd, 0, 0, SYNC_FILE_RANGE_WAIT_BEFORE |
+ SYNC_FILE_RANGE_WRITE |
+ SYNC_FILE_RANGE_WAIT_AFTER);
+#endif
+
+#ifdef fsync_no_flush
+ return fsync_no_flush(fd);
+#endif
+
+ errno = ENOSYS;
+ return -1;
+
+ case FSYNC_HARDWARE_FLUSH:
+ /*
+ * On macOS, a special fcntl is required to really flush the
+ * caches within the storage controller. As of this writing,
+ * this is a very expensive operation on Apple SSDs.
+ */
+#ifdef __APPLE__
+ return fcntl(fd, F_FULLFSYNC);
+#else
+ return fsync_loop(fd);
+#endif
+ default:
+ BUG("unexpected git_fsync(%d) call", action);
+ }
+}
+
static int warn_if_unremovable(const char *op, const char *file, int rc)
{
int err;
diff --git a/write-or-die.c b/write-or-die.c
index a3d5784cec9..9faa5f9f563 100644
--- a/write-or-die.c
+++ b/write-or-die.c
@@ -62,10 +62,13 @@ void fsync_or_die(int fd, const char *msg)
use_fsync = git_env_bool("GIT_TEST_FSYNC", 1);
if (!use_fsync)
return;
- while (fsync(fd) < 0) {
- if (errno != EINTR)
- die_errno("fsync error on '%s'", msg);
- }
+
+ if (fsync_method == FSYNC_METHOD_WRITEOUT_ONLY &&
+ git_fsync(fd, FSYNC_WRITEOUT_ONLY) >= 0)
+ return;
+
+ if (git_fsync(fd, FSYNC_HARDWARE_FLUSH) < 0)
+ die_errno("fsync error on '%s'", msg);
}
void write_or_die(int fd, const void *buf, size_t count)
--
gitgitgadget
^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v5 3/5] core.fsync: introduce granular fsync control
2022-03-09 23:03 ` [PATCH v5 0/5] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget
2022-03-09 23:03 ` [PATCH v5 1/5] wrapper: move inclusion of CSPRNG headers the wrapper.c file Neeraj Singh via GitGitGadget
2022-03-09 23:03 ` [PATCH v5 2/5] core.fsyncmethod: add writeout-only mode Neeraj Singh via GitGitGadget
@ 2022-03-09 23:03 ` Neeraj Singh via GitGitGadget
2022-03-10 0:21 ` Junio C Hamano
2022-03-09 23:03 ` [PATCH v5 4/5] core.fsync: new option to harden the index Neeraj Singh via GitGitGadget
` (6 subsequent siblings)
9 siblings, 1 reply; 122+ messages in thread
From: Neeraj Singh via GitGitGadget @ 2022-03-09 23:03 UTC (permalink / raw)
To: git
Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps, sandals,
Neeraj K. Singh, Neeraj Singh
From: Neeraj Singh <neerajsi@microsoft.com>
This commit introduces the `core.fsync` configuration
knob which can be used to control how components of the
repository are made durable on disk.
This setting allows future extensibility of the list of
syncable components:
* We issue a warning rather than an error for unrecognized
components, so new configs can be used with old Git versions.
* We support negation, so users can choose one of the aggregate
options and then remove components that they don't want.
Aggregate options are defined in a later patch in this series.
This also supports the common request of doing absolutely no
fysncing with the `core.fsync=none` value, which is expected
to make the test suite faster.
Complete documentation for the new setting is included in a later patch
in the series so that it can be reviewed in final form.
Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
---
Documentation/config/core.txt | 8 ----
builtin/fast-import.c | 2 +-
builtin/index-pack.c | 4 +-
builtin/pack-objects.c | 24 ++++++++----
bulk-checkin.c | 5 ++-
cache.h | 30 +++++++++++++-
commit-graph.c | 3 +-
config.c | 73 ++++++++++++++++++++++++++++++++++-
csum-file.c | 5 ++-
csum-file.h | 3 +-
environment.c | 2 +-
midx.c | 3 +-
object-file.c | 3 +-
pack-bitmap-write.c | 3 +-
pack-write.c | 13 ++++---
read-cache.c | 2 +-
16 files changed, 144 insertions(+), 39 deletions(-)
diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
index dbb134f7136..74399072843 100644
--- a/Documentation/config/core.txt
+++ b/Documentation/config/core.txt
@@ -556,14 +556,6 @@ core.fsyncMethod::
filesystem and storage hardware, data added to the repository may not be
durable in the event of a system crash. This is the default mode on macOS.
-core.fsyncObjectFiles::
- This boolean will enable 'fsync()' when writing object files.
-+
-This is a total waste of time and effort on a filesystem that orders
-data writes properly, but can be useful for filesystems that do not use
-journalling (traditional UNIX filesystems) or that only journal metadata
-and not file contents (OS X's HFS+, or Linux ext3 with "data=writeback").
-
core.preloadIndex::
Enable parallel index preload for operations like 'git diff'
+
diff --git a/builtin/fast-import.c b/builtin/fast-import.c
index b7105fcad9b..f2c036a8955 100644
--- a/builtin/fast-import.c
+++ b/builtin/fast-import.c
@@ -865,7 +865,7 @@ static void end_packfile(void)
struct tag *t;
close_pack_windows(pack_data);
- finalize_hashfile(pack_file, cur_pack_oid.hash, 0);
+ finalize_hashfile(pack_file, cur_pack_oid.hash, FSYNC_COMPONENT_PACK, 0);
fixup_pack_header_footer(pack_data->pack_fd, pack_data->hash,
pack_data->pack_name, object_count,
cur_pack_oid.hash, pack_size);
diff --git a/builtin/index-pack.c b/builtin/index-pack.c
index c45273de3b1..c5f12f14df5 100644
--- a/builtin/index-pack.c
+++ b/builtin/index-pack.c
@@ -1290,7 +1290,7 @@ static void conclude_pack(int fix_thin_pack, const char *curr_pack, unsigned cha
nr_objects - nr_objects_initial);
stop_progress_msg(&progress, msg.buf);
strbuf_release(&msg);
- finalize_hashfile(f, tail_hash, 0);
+ finalize_hashfile(f, tail_hash, FSYNC_COMPONENT_PACK, 0);
hashcpy(read_hash, pack_hash);
fixup_pack_header_footer(output_fd, pack_hash,
curr_pack, nr_objects,
@@ -1512,7 +1512,7 @@ static void final(const char *final_pack_name, const char *curr_pack_name,
if (!from_stdin) {
close(input_fd);
} else {
- fsync_or_die(output_fd, curr_pack_name);
+ fsync_component_or_die(FSYNC_COMPONENT_PACK, output_fd, curr_pack_name);
err = close(output_fd);
if (err)
die_errno(_("error while closing pack file"));
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index 178e611f09d..c14fee8e99f 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -1199,16 +1199,26 @@ static void write_pack_file(void)
display_progress(progress_state, written);
}
- /*
- * Did we write the wrong # entries in the header?
- * If so, rewrite it like in fast-import
- */
if (pack_to_stdout) {
- finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_CLOSE);
+ /*
+ * We never fsync when writing to stdout since we may
+ * not be writing to an actual pack file. For instance,
+ * the upload-pack code passes a pipe here. Calling
+ * fsync on a pipe results in unnecessary
+ * synchronization with the reader on some platforms.
+ */
+ finalize_hashfile(f, hash, FSYNC_COMPONENT_NONE,
+ CSUM_HASH_IN_STREAM | CSUM_CLOSE);
} else if (nr_written == nr_remaining) {
- finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
+ finalize_hashfile(f, hash, FSYNC_COMPONENT_PACK,
+ CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
} else {
- int fd = finalize_hashfile(f, hash, 0);
+ /*
+ * If we wrote the wrong number of entries in the
+ * header, rewrite it like in fast-import.
+ */
+
+ int fd = finalize_hashfile(f, hash, FSYNC_COMPONENT_PACK, 0);
fixup_pack_header_footer(fd, hash, pack_tmp_name,
nr_written, hash, offset);
close(fd);
diff --git a/bulk-checkin.c b/bulk-checkin.c
index 8785b2ac806..a2cf9dcbc8d 100644
--- a/bulk-checkin.c
+++ b/bulk-checkin.c
@@ -53,9 +53,10 @@ static void finish_bulk_checkin(struct bulk_checkin_state *state)
unlink(state->pack_tmp_name);
goto clear_exit;
} else if (state->nr_written == 1) {
- finalize_hashfile(state->f, hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
+ finalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK,
+ CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
} else {
- int fd = finalize_hashfile(state->f, hash, 0);
+ int fd = finalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK, 0);
fixup_pack_header_footer(fd, hash, state->pack_tmp_name,
state->nr_written, hash,
state->offset);
diff --git a/cache.h b/cache.h
index 82f0194a3dd..a5eaa60a7a8 100644
--- a/cache.h
+++ b/cache.h
@@ -993,8 +993,27 @@ void reset_shared_repository(void);
extern int read_replace_refs;
extern char *git_replace_ref_base;
-extern int fsync_object_files;
-extern int use_fsync;
+/*
+ * These values are used to help identify parts of a repository to fsync.
+ * FSYNC_COMPONENT_NONE identifies data that will not be a persistent part of the
+ * repository and so shouldn't be fsynced.
+ */
+enum fsync_component {
+ FSYNC_COMPONENT_NONE,
+ FSYNC_COMPONENT_LOOSE_OBJECT = 1 << 0,
+ FSYNC_COMPONENT_PACK = 1 << 1,
+ FSYNC_COMPONENT_PACK_METADATA = 1 << 2,
+ FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3,
+};
+
+#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \
+ FSYNC_COMPONENT_PACK_METADATA | \
+ FSYNC_COMPONENT_COMMIT_GRAPH)
+
+/*
+ * A bitmask indicating which components of the repo should be fsynced.
+ */
+extern enum fsync_component fsync_components;
enum fsync_method {
FSYNC_METHOD_FSYNC,
@@ -1002,6 +1021,7 @@ enum fsync_method {
};
extern enum fsync_method fsync_method;
+extern int use_fsync;
extern int core_preload_index;
extern int precomposed_unicode;
extern int protect_hfs;
@@ -1708,6 +1728,12 @@ int copy_file_with_time(const char *dst, const char *src, int mode);
void write_or_die(int fd, const void *buf, size_t count);
void fsync_or_die(int fd, const char *);
+static inline void fsync_component_or_die(enum fsync_component component, int fd, const char *msg)
+{
+ if (fsync_components & component)
+ fsync_or_die(fd, msg);
+}
+
ssize_t read_in_full(int fd, void *buf, size_t count);
ssize_t write_in_full(int fd, const void *buf, size_t count);
ssize_t pread_in_full(int fd, void *buf, size_t count, off_t offset);
diff --git a/commit-graph.c b/commit-graph.c
index 265c010122e..64897f57d9f 100644
--- a/commit-graph.c
+++ b/commit-graph.c
@@ -1942,7 +1942,8 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)
}
close_commit_graph(ctx->r->objects);
- finalize_hashfile(f, file_hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC);
+ finalize_hashfile(f, file_hash, FSYNC_COMPONENT_COMMIT_GRAPH,
+ CSUM_HASH_IN_STREAM | CSUM_FSYNC);
free_chunkfile(cf);
if (ctx->split) {
diff --git a/config.c b/config.c
index f3ff80b01c9..51a35715642 100644
--- a/config.c
+++ b/config.c
@@ -1323,6 +1323,70 @@ static int git_parse_maybe_bool_text(const char *value)
return -1;
}
+static const struct fsync_component_entry {
+ const char *name;
+ enum fsync_component component_bits;
+} fsync_component_table[] = {
+ { "loose-object", FSYNC_COMPONENT_LOOSE_OBJECT },
+ { "pack", FSYNC_COMPONENT_PACK },
+ { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA },
+ { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH },
+};
+
+static enum fsync_component parse_fsync_components(const char *var, const char *string)
+{
+ enum fsync_component output = 0;
+
+ if (!strcmp(string, "none"))
+ return FSYNC_COMPONENT_NONE;
+
+ while (string) {
+ int i;
+ size_t len;
+ const char *ep;
+ int negated = 0;
+ int found = 0;
+
+ string = string + strspn(string, ", \t\n\r");
+ ep = strchrnul(string, ',');
+ len = ep - string;
+
+ if (*string == '-') {
+ negated = 1;
+ string++;
+ len--;
+ if (!len)
+ warning(_("invalid value for variable %s"), var);
+ }
+
+ if (!len)
+ break;
+
+ for (i = 0; i < ARRAY_SIZE(fsync_component_table); ++i) {
+ const struct fsync_component_entry *entry = &fsync_component_table[i];
+
+ if (strncmp(entry->name, string, len))
+ continue;
+
+ found = 1;
+ if (negated)
+ output &= ~entry->component_bits;
+ else
+ output |= entry->component_bits;
+ }
+
+ if (!found) {
+ char *component = xstrndup(string, len);
+ warning(_("ignoring unknown core.fsync component '%s'"), component);
+ free(component);
+ }
+
+ string = ep;
+ }
+
+ return output;
+}
+
int git_parse_maybe_bool(const char *value)
{
int v = git_parse_maybe_bool_text(value);
@@ -1600,6 +1664,13 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
return 0;
}
+ if (!strcmp(var, "core.fsync")) {
+ if (!value)
+ return config_error_nonbool(var);
+ fsync_components = parse_fsync_components(var, value);
+ return 0;
+ }
+
if (!strcmp(var, "core.fsyncmethod")) {
if (!value)
return config_error_nonbool(var);
@@ -1613,7 +1684,7 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
}
if (!strcmp(var, "core.fsyncobjectfiles")) {
- fsync_object_files = git_config_bool(var, value);
+ warning(_("core.fsyncobjectfiles is deprecated; use core.fsync instead"));
return 0;
}
diff --git a/csum-file.c b/csum-file.c
index 26e8a6df44e..59ef3398ca2 100644
--- a/csum-file.c
+++ b/csum-file.c
@@ -58,7 +58,8 @@ static void free_hashfile(struct hashfile *f)
free(f);
}
-int finalize_hashfile(struct hashfile *f, unsigned char *result, unsigned int flags)
+int finalize_hashfile(struct hashfile *f, unsigned char *result,
+ enum fsync_component component, unsigned int flags)
{
int fd;
@@ -69,7 +70,7 @@ int finalize_hashfile(struct hashfile *f, unsigned char *result, unsigned int fl
if (flags & CSUM_HASH_IN_STREAM)
flush(f, f->buffer, the_hash_algo->rawsz);
if (flags & CSUM_FSYNC)
- fsync_or_die(f->fd, f->name);
+ fsync_component_or_die(component, f->fd, f->name);
if (flags & CSUM_CLOSE) {
if (close(f->fd))
die_errno("%s: sha1 file error on close", f->name);
diff --git a/csum-file.h b/csum-file.h
index 291215b34eb..0d29f528fbc 100644
--- a/csum-file.h
+++ b/csum-file.h
@@ -1,6 +1,7 @@
#ifndef CSUM_FILE_H
#define CSUM_FILE_H
+#include "cache.h"
#include "hash.h"
struct progress;
@@ -38,7 +39,7 @@ int hashfile_truncate(struct hashfile *, struct hashfile_checkpoint *);
struct hashfile *hashfd(int fd, const char *name);
struct hashfile *hashfd_check(const char *name);
struct hashfile *hashfd_throughput(int fd, const char *name, struct progress *tp);
-int finalize_hashfile(struct hashfile *, unsigned char *, unsigned int);
+int finalize_hashfile(struct hashfile *, unsigned char *, enum fsync_component, unsigned int);
void hashwrite(struct hashfile *, const void *, unsigned int);
void hashflush(struct hashfile *f);
void crc32_begin(struct hashfile *);
diff --git a/environment.c b/environment.c
index 3e3620d759f..378424b9af5 100644
--- a/environment.c
+++ b/environment.c
@@ -42,9 +42,9 @@ const char *git_attributes_file;
const char *git_hooks_path;
int zlib_compression_level = Z_BEST_SPEED;
int pack_compression_level = Z_DEFAULT_COMPRESSION;
-int fsync_object_files;
int use_fsync = -1;
enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT;
+enum fsync_component fsync_components = FSYNC_COMPONENTS_DEFAULT;
size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE;
size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT;
size_t delta_base_cache_limit = 96 * 1024 * 1024;
diff --git a/midx.c b/midx.c
index 865170bad05..107365d2114 100644
--- a/midx.c
+++ b/midx.c
@@ -1438,7 +1438,8 @@ static int write_midx_internal(const char *object_dir,
write_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs);
write_chunkfile(cf, &ctx);
- finalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);
+ finalize_hashfile(f, midx_hash, FSYNC_COMPONENT_PACK_METADATA,
+ CSUM_FSYNC | CSUM_HASH_IN_STREAM);
free_chunkfile(cf);
if (flags & MIDX_WRITE_REV_INDEX &&
diff --git a/object-file.c b/object-file.c
index 03bd6a3baf3..c9de912faca 100644
--- a/object-file.c
+++ b/object-file.c
@@ -1850,8 +1850,7 @@ int hash_object_file(const struct git_hash_algo *algo, const void *buf,
static void close_loose_object(int fd)
{
if (!the_repository->objects->odb->will_destroy) {
- if (fsync_object_files)
- fsync_or_die(fd, "loose object file");
+ fsync_component_or_die(FSYNC_COMPONENT_LOOSE_OBJECT, fd, "loose object file");
}
if (close(fd) != 0)
diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c
index cab3eaa2acd..cf681547f2e 100644
--- a/pack-bitmap-write.c
+++ b/pack-bitmap-write.c
@@ -719,7 +719,8 @@ void bitmap_writer_finish(struct pack_idx_entry **index,
if (options & BITMAP_OPT_HASH_CACHE)
write_hash_cache(f, index, index_nr);
- finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
+ finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA,
+ CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
if (adjust_shared_perm(tmp_file.buf))
die_errno("unable to make temporary bitmap file readable");
diff --git a/pack-write.c b/pack-write.c
index a5846f3a346..51812cb1299 100644
--- a/pack-write.c
+++ b/pack-write.c
@@ -159,9 +159,9 @@ const char *write_idx_file(const char *index_name, struct pack_idx_entry **objec
}
hashwrite(f, sha1, the_hash_algo->rawsz);
- finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE |
- ((opts->flags & WRITE_IDX_VERIFY)
- ? 0 : CSUM_FSYNC));
+ finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA,
+ CSUM_HASH_IN_STREAM | CSUM_CLOSE |
+ ((opts->flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC));
return index_name;
}
@@ -281,8 +281,9 @@ const char *write_rev_file_order(const char *rev_name,
if (rev_name && adjust_shared_perm(rev_name) < 0)
die(_("failed to make %s readable"), rev_name);
- finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE |
- ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC));
+ finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA,
+ CSUM_HASH_IN_STREAM | CSUM_CLOSE |
+ ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC));
return rev_name;
}
@@ -390,7 +391,7 @@ void fixup_pack_header_footer(int pack_fd,
the_hash_algo->final_fn(partial_pack_hash, &old_hash_ctx);
the_hash_algo->final_fn(new_pack_hash, &new_hash_ctx);
write_or_die(pack_fd, new_pack_hash, the_hash_algo->rawsz);
- fsync_or_die(pack_fd, pack_name);
+ fsync_component_or_die(FSYNC_COMPONENT_PACK, pack_fd, pack_name);
}
char *index_pack_lockfile(int ip_out, int *is_well_formed)
diff --git a/read-cache.c b/read-cache.c
index 79b9b99ebf7..df869691fd4 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -3089,7 +3089,7 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile,
return -1;
}
- finalize_hashfile(f, istate->oid.hash, CSUM_HASH_IN_STREAM);
+ finalize_hashfile(f, istate->oid.hash, FSYNC_COMPONENT_NONE, CSUM_HASH_IN_STREAM);
if (close_tempfile_gently(tempfile)) {
error(_("could not close '%s'"), get_tempfile_path(tempfile));
return -1;
--
gitgitgadget
^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v5 4/5] core.fsync: new option to harden the index
2022-03-09 23:03 ` [PATCH v5 0/5] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget
` (2 preceding siblings ...)
2022-03-09 23:03 ` [PATCH v5 3/5] core.fsync: introduce granular fsync control Neeraj Singh via GitGitGadget
@ 2022-03-09 23:03 ` Neeraj Singh via GitGitGadget
2022-03-09 23:03 ` [PATCH v5 5/5] core.fsync: documentation and user-friendly aggregate options Neeraj Singh via GitGitGadget
` (5 subsequent siblings)
9 siblings, 0 replies; 122+ messages in thread
From: Neeraj Singh via GitGitGadget @ 2022-03-09 23:03 UTC (permalink / raw)
To: git
Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps, sandals,
Neeraj K. Singh, Neeraj Singh
From: Neeraj Singh <neerajsi@microsoft.com>
This commit introduces the new ability for the user to harden
the index. In the event of a system crash, the index must be
durable for the user to actually find a file that has been added
to the repo and then deleted from the working tree.
We use the presence of the COMMIT_LOCK flag and absence of the
alternate_index_output as a proxy for determining whether we're
updating the persistent index of the repo or some temporary
index. We don't sync these temporary indexes.
Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
---
cache.h | 1 +
config.c | 1 +
read-cache.c | 19 +++++++++++++------
3 files changed, 15 insertions(+), 6 deletions(-)
diff --git a/cache.h b/cache.h
index a5eaa60a7a8..3fefe8f6f60 100644
--- a/cache.h
+++ b/cache.h
@@ -1004,6 +1004,7 @@ enum fsync_component {
FSYNC_COMPONENT_PACK = 1 << 1,
FSYNC_COMPONENT_PACK_METADATA = 1 << 2,
FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3,
+ FSYNC_COMPONENT_INDEX = 1 << 4,
};
#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \
diff --git a/config.c b/config.c
index 51a35715642..bb44f6f506d 100644
--- a/config.c
+++ b/config.c
@@ -1331,6 +1331,7 @@ static const struct fsync_component_entry {
{ "pack", FSYNC_COMPONENT_PACK },
{ "pack-metadata", FSYNC_COMPONENT_PACK_METADATA },
{ "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH },
+ { "index", FSYNC_COMPONENT_INDEX },
};
static enum fsync_component parse_fsync_components(const char *var, const char *string)
diff --git a/read-cache.c b/read-cache.c
index df869691fd4..7683b679258 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -2842,7 +2842,7 @@ static int record_ieot(void)
* rely on it.
*/
static int do_write_index(struct index_state *istate, struct tempfile *tempfile,
- int strip_extensions)
+ int strip_extensions, unsigned flags)
{
uint64_t start = getnanotime();
struct hashfile *f;
@@ -2856,6 +2856,7 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile,
struct strbuf previous_name_buf = STRBUF_INIT, *previous_name;
int drop_cache_tree = istate->drop_cache_tree;
off_t offset;
+ int csum_fsync_flag;
int ieot_entries = 1;
struct index_entry_offset_table *ieot = NULL;
int nr, nr_threads;
@@ -3089,7 +3090,13 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile,
return -1;
}
- finalize_hashfile(f, istate->oid.hash, FSYNC_COMPONENT_NONE, CSUM_HASH_IN_STREAM);
+ csum_fsync_flag = 0;
+ if (!alternate_index_output && (flags & COMMIT_LOCK))
+ csum_fsync_flag = CSUM_FSYNC;
+
+ finalize_hashfile(f, istate->oid.hash, FSYNC_COMPONENT_INDEX,
+ CSUM_HASH_IN_STREAM | csum_fsync_flag);
+
if (close_tempfile_gently(tempfile)) {
error(_("could not close '%s'"), get_tempfile_path(tempfile));
return -1;
@@ -3144,7 +3151,7 @@ static int do_write_locked_index(struct index_state *istate, struct lock_file *l
*/
trace2_region_enter_printf("index", "do_write_index", the_repository,
"%s", get_lock_file_path(lock));
- ret = do_write_index(istate, lock->tempfile, 0);
+ ret = do_write_index(istate, lock->tempfile, 0, flags);
trace2_region_leave_printf("index", "do_write_index", the_repository,
"%s", get_lock_file_path(lock));
@@ -3238,7 +3245,7 @@ static int clean_shared_index_files(const char *current_hex)
}
static int write_shared_index(struct index_state *istate,
- struct tempfile **temp)
+ struct tempfile **temp, unsigned flags)
{
struct split_index *si = istate->split_index;
int ret, was_full = !istate->sparse_index;
@@ -3248,7 +3255,7 @@ static int write_shared_index(struct index_state *istate,
trace2_region_enter_printf("index", "shared/do_write_index",
the_repository, "%s", get_tempfile_path(*temp));
- ret = do_write_index(si->base, *temp, 1);
+ ret = do_write_index(si->base, *temp, 1, flags);
trace2_region_leave_printf("index", "shared/do_write_index",
the_repository, "%s", get_tempfile_path(*temp));
@@ -3357,7 +3364,7 @@ int write_locked_index(struct index_state *istate, struct lock_file *lock,
ret = do_write_locked_index(istate, lock, flags);
goto out;
}
- ret = write_shared_index(istate, &temp);
+ ret = write_shared_index(istate, &temp, flags);
saved_errno = errno;
if (is_tempfile_active(temp))
--
gitgitgadget
^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v5 5/5] core.fsync: documentation and user-friendly aggregate options
2022-03-09 23:03 ` [PATCH v5 0/5] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget
` (3 preceding siblings ...)
2022-03-09 23:03 ` [PATCH v5 4/5] core.fsync: new option to harden the index Neeraj Singh via GitGitGadget
@ 2022-03-09 23:03 ` Neeraj Singh via GitGitGadget
2022-03-10 9:53 ` Future-proofed syncing of refs Patrick Steinhardt
` (4 subsequent siblings)
9 siblings, 0 replies; 122+ messages in thread
From: Neeraj Singh via GitGitGadget @ 2022-03-09 23:03 UTC (permalink / raw)
To: git
Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps, sandals,
Neeraj K. Singh, Neeraj Singh
From: Neeraj Singh <neerajsi@microsoft.com>
This commit adds aggregate options for the core.fsync setting that are
more user-friendly. These options are specified in terms of 'levels of
safety', indicating which Git operations are considered to be sync
points for durability.
The new documentation is also included here in its entirety for ease of
review.
Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
---
Documentation/config/core.txt | 36 +++++++++++++++++++++++++++++++++++
cache.h | 23 +++++++++++++++++++---
config.c | 6 ++++++
3 files changed, 62 insertions(+), 3 deletions(-)
diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
index 74399072843..973805e8a98 100644
--- a/Documentation/config/core.txt
+++ b/Documentation/config/core.txt
@@ -547,6 +547,42 @@ core.whitespace::
is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent`
errors. The default tab width is 8. Allowed values are 1 to 63.
+core.fsync::
+ A comma-separated list of parts of the repository which should be
+ hardened via the core.fsyncMethod when created or modified. You can
+ disable hardening of any component by prefixing it with a '-'. Later
+ items take precedence over earlier ones in the comma-separated list.
+ For example, `core.fsync=all,-pack-metadata` means "harden everything
+ except pack metadata." Items that are not hardened may be lost in the
+ event of an unclean system shutdown. Unless you have special
+ requirements, it is recommended that you leave this option as default
+ or pick one of `committed`, `added`, or `all`.
++
+* `none` disables fsync completely. This value must be specified alone.
+* `loose-object` hardens objects added to the repo in loose-object form.
+* `pack` hardens objects added to the repo in packfile form.
+* `pack-metadata` hardens packfile bitmaps and indexes.
+* `commit-graph` hardens the commit graph file.
+* `index` hardens the index when it is modified.
+* `objects` is an aggregate option that is equivalent to
+ `loose-object,pack`.
+* `derived-metadata` is an aggregate option that is equivalent to
+ `pack-metadata,commit-graph`.
+* `default` is an aggregate option that is equivalent to
+ `objects,derived-metadata,-loose-object`. This mode is enabled by default.
+ It has good performance, but risks losing recent work if the system shuts
+ down uncleanly, since commits, trees, and blobs in loose-object form may be
+ lost.
+* `committed` is an aggregate option that is currently equivalent to
+ `objects`. This mode sacrifices some performance to ensure that all work
+ that is committed to the repository with `git commit` or similar commands
+ is preserved.
+* `added` is an aggregate option that is currently equivalent to
+ `committed,index`. This mode sacrifices additional performance to
+ ensure that the results of commands like `git add` and similar operations
+ are preserved.
+* `all` is an aggregate option that syncs all individual components above.
+
core.fsyncMethod::
A value indicating the strategy Git will use to harden repository data
using fsync and related primitives.
diff --git a/cache.h b/cache.h
index 3fefe8f6f60..833f0236e68 100644
--- a/cache.h
+++ b/cache.h
@@ -1007,9 +1007,26 @@ enum fsync_component {
FSYNC_COMPONENT_INDEX = 1 << 4,
};
-#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \
- FSYNC_COMPONENT_PACK_METADATA | \
- FSYNC_COMPONENT_COMMIT_GRAPH)
+#define FSYNC_COMPONENTS_OBJECTS (FSYNC_COMPONENT_LOOSE_OBJECT | \
+ FSYNC_COMPONENT_PACK)
+
+#define FSYNC_COMPONENTS_DERIVED_METADATA (FSYNC_COMPONENT_PACK_METADATA | \
+ FSYNC_COMPONENT_COMMIT_GRAPH)
+
+#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENTS_OBJECTS | \
+ FSYNC_COMPONENTS_DERIVED_METADATA | \
+ ~FSYNC_COMPONENT_LOOSE_OBJECT)
+
+#define FSYNC_COMPONENTS_COMMITTED (FSYNC_COMPONENTS_OBJECTS)
+
+#define FSYNC_COMPONENTS_ADDED (FSYNC_COMPONENTS_COMMITTED | \
+ FSYNC_COMPONENT_INDEX)
+
+#define FSYNC_COMPONENTS_ALL (FSYNC_COMPONENT_LOOSE_OBJECT | \
+ FSYNC_COMPONENT_PACK | \
+ FSYNC_COMPONENT_PACK_METADATA | \
+ FSYNC_COMPONENT_COMMIT_GRAPH | \
+ FSYNC_COMPONENT_INDEX)
/*
* A bitmask indicating which components of the repo should be fsynced.
diff --git a/config.c b/config.c
index bb44f6f506d..3976ec74fd4 100644
--- a/config.c
+++ b/config.c
@@ -1332,6 +1332,12 @@ static const struct fsync_component_entry {
{ "pack-metadata", FSYNC_COMPONENT_PACK_METADATA },
{ "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH },
{ "index", FSYNC_COMPONENT_INDEX },
+ { "objects", FSYNC_COMPONENTS_OBJECTS },
+ { "derived-metadata", FSYNC_COMPONENTS_DERIVED_METADATA },
+ { "default", FSYNC_COMPONENTS_DEFAULT },
+ { "committed", FSYNC_COMPONENTS_COMMITTED },
+ { "added", FSYNC_COMPONENTS_ADDED },
+ { "all", FSYNC_COMPONENTS_ALL },
};
static enum fsync_component parse_fsync_components(const char *var, const char *string)
--
gitgitgadget
^ permalink raw reply related [flat|nested] 122+ messages in thread
* Re: [PATCH v5 1/5] wrapper: move inclusion of CSPRNG headers the wrapper.c file
2022-03-09 23:03 ` [PATCH v5 1/5] wrapper: move inclusion of CSPRNG headers the wrapper.c file Neeraj Singh via GitGitGadget
@ 2022-03-09 23:29 ` Junio C Hamano
2022-03-10 1:21 ` Neeraj Singh
2022-03-10 1:26 ` brian m. carlson
1 sibling, 1 reply; 122+ messages in thread
From: Junio C Hamano @ 2022-03-09 23:29 UTC (permalink / raw)
To: Neeraj Singh via GitGitGadget
Cc: git, rsbecker, bagasdotme, newren, avarab, nksingh85, ps, sandals,
Neeraj K. Singh
"Neeraj Singh via GitGitGadget" <gitgitgadget@gmail.com> writes:
> Including NTSecAPI.h in git-compat-util.h causes build errors in any
> other file that includes winternl.h. That file was included in order to
> get access to the RtlGenRandom cryptographically secure PRNG. This
> change scopes the inclusion of all PRNG headers to just the wrapper.c
> file, which is the only place it is really needed.
It is true that wrapper.c is the only thing that needs these headers
included as part of its implementation detail of csprng_bytes(), and
I think I very much like this change for that reason.
Having said that, if it true that including these two header files
in the same file will lead to compilation failure? That sounds like
either (1) they represent two mutually incompatible APIs that will
cause breakage when code that use them, perhaps in two separate
files to avoid compilation failures, are linked together, or (2)
these system header files are simply broken. Or something else?
> -/*
> - * Including the appropriate header file for RtlGenRandom causes MSVC to see a
> - * redefinition of types in an incompatible way when including headers below.
> - */
> -#undef HAVE_RTLGENRANDOM
This comment hints it is more like (1) above? A type used in one
part of the system is defined differently in other parts of the
system? I cannot imagine anything but bad things happen when a
piece of code uses one definition of the type to declare a function,
and another piece of code uses the other definition of the same type
to declare a variable and passes it as a parameter to that function.
I do not know this patch makes the situation worse, and I am not a
Windows person with boxes to dig deeper to begin with. Hence I do
not mind the change itself, but justifying the change primarily as a
workaround for some obscure header type clashes on a single system
leaves a bad taste. If the first sentence weren't there, I wouldn't
have spent this many minutes wondering if this is a good change ;-)
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v5 2/5] core.fsyncmethod: add writeout-only mode
2022-03-09 23:03 ` [PATCH v5 2/5] core.fsyncmethod: add writeout-only mode Neeraj Singh via GitGitGadget
@ 2022-03-09 23:48 ` Junio C Hamano
0 siblings, 0 replies; 122+ messages in thread
From: Junio C Hamano @ 2022-03-09 23:48 UTC (permalink / raw)
To: Neeraj Singh via GitGitGadget
Cc: git, rsbecker, bagasdotme, newren, avarab, nksingh85, ps, sandals,
Neeraj K. Singh
"Neeraj Singh via GitGitGadget" <gitgitgadget@gmail.com> writes:
> From: Neeraj Singh <neerajsi@microsoft.com>
>
> This commit introduces the `core.fsyncMethod` configuration
> knob, which can currently be set to `fsync` or `writeout-only`.
>
> The new writeout-only mode attempts to tell the operating system to
> flush its in-memory page cache to the storage hardware without issuing a
> CACHE_FLUSH command to the storage controller.
>
> Writeout-only fsync is significantly faster than a vanilla fsync on
> common hardware, since data is written to a disk-side cache rather than
> all the way to a durable medium. Later changes in this patch series will
> take advantage of this primitive to implement batching of hardware
> flushes.
>
> When git_fsync is called with FSYNC_WRITEOUT_ONLY, it may fail and the
> caller is expected to do an ordinary fsync as needed.
>
> On Apple platforms, the fsync system call does not issue a CACHE_FLUSH
> directive to the storage controller. This change updates fsync to do
> fcntl(F_FULLFSYNC) to make fsync actually durable. We maintain parity
> with existing behavior on Apple platforms by setting the default value
> of the new core.fsyncMethod option.
>
> Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
> ---
OK. This seems to be quite reasonable in that the pieces of code
that use fsync_or_die() do not have to change at all, and all of
them will keep behaving the same way. In other words, the "how" of
fsync is very much well isolated.
Nice.
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v5 3/5] core.fsync: introduce granular fsync control
2022-03-09 23:03 ` [PATCH v5 3/5] core.fsync: introduce granular fsync control Neeraj Singh via GitGitGadget
@ 2022-03-10 0:21 ` Junio C Hamano
2022-03-10 2:53 ` Neeraj Singh
0 siblings, 1 reply; 122+ messages in thread
From: Junio C Hamano @ 2022-03-10 0:21 UTC (permalink / raw)
To: Neeraj Singh via GitGitGadget
Cc: git, rsbecker, bagasdotme, newren, avarab, nksingh85, ps, sandals,
Neeraj K. Singh
"Neeraj Singh via GitGitGadget" <gitgitgadget@gmail.com> writes:
> +/*
> + * These values are used to help identify parts of a repository to fsync.
> + * FSYNC_COMPONENT_NONE identifies data that will not be a persistent part of the
> + * repository and so shouldn't be fsynced.
> + */
> +enum fsync_component {
> + FSYNC_COMPONENT_NONE,
> + FSYNC_COMPONENT_LOOSE_OBJECT = 1 << 0,
> + FSYNC_COMPONENT_PACK = 1 << 1,
> + FSYNC_COMPONENT_PACK_METADATA = 1 << 2,
> + FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3,
> +};
OK, so the idea is that Patrick's "we need to fsync refs" will be
done by adding a new component to this list, and sprinkling a call
to fsync_component_or_die() in the code of ref-files backend?
I am wondering if fsync_or_die() interface is abstracted well
enough, or we need things like "the fd is inside this directory; in
addition to doing the fsync of the fd, please sync the parent
directory as well" support before we start adding more components
(if there is such a need, perhaps it comes before this step).
> +#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \
> + FSYNC_COMPONENT_PACK_METADATA | \
> + FSYNC_COMPONENT_COMMIT_GRAPH)
IOW, everything other than loose object, which already has a
separate core.fsyncObjectFiles knob to loosen. Everything else we
currently sync unconditionally and the default keeps that
arrangement?
> +static inline void fsync_component_or_die(enum fsync_component component, int fd, const char *msg)
> +{
> + if (fsync_components & component)
> + fsync_or_die(fd, msg);
> +}
Do we have a compelling reason to have this as a static inline
function? We are talking about concluding an I/O operation and
I doubt there is a good performance argument for it.
> +static const struct fsync_component_entry {
> + const char *name;
> + enum fsync_component component_bits;
> +} fsync_component_table[] = {
thing[] is an array of "thing" (and thing[4] is the "fourth" such
thing), but this is not an array of a table (it is a name-to-bit
mapping).
I wonder if this array works without "_table" suffix in its name.
> + { "loose-object", FSYNC_COMPONENT_LOOSE_OBJECT },
> + { "pack", FSYNC_COMPONENT_PACK },
> + { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA },
> + { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH },
> +};
> +
> +static enum fsync_component parse_fsync_components(const char *var, const char *string)
> +{
> + enum fsync_component output = 0;
> +
> + if (!strcmp(string, "none"))
> + return FSYNC_COMPONENT_NONE;
> +
> + while (string) {
> + int i;
> + size_t len;
> + const char *ep;
> + int negated = 0;
> + int found = 0;
> +
> + string = string + strspn(string, ", \t\n\r");
> + ep = strchrnul(string, ',');
> + len = ep - string;
> +
> + if (*string == '-') {
> + negated = 1;
> + string++;
> + len--;
> + if (!len)
> + warning(_("invalid value for variable %s"), var);
> + }
> +
> + if (!len)
> + break;
> +
> + for (i = 0; i < ARRAY_SIZE(fsync_component_table); ++i) {
> + const struct fsync_component_entry *entry = &fsync_component_table[i];
> +
> + if (strncmp(entry->name, string, len))
> + continue;
> +
> + found = 1;
> + if (negated)
> + output &= ~entry->component_bits;
> + else
> + output |= entry->component_bits;
> + }
> +
> + if (!found) {
> + char *component = xstrndup(string, len);
> + warning(_("ignoring unknown core.fsync component '%s'"), component);
> + free(component);
> + }
> +
> + string = ep;
> + }
> +
> + return output;
> +}
Hmph. I would have expected, with built-in default of
pack,pack-metadata,commit-graph,
- "none,pack" would choose only "pack" by first clearing the
built-in default (or whatever was set in configuration files that
are lower precedence than what we are reading) and then OR'ing
the "pack" bit in.
- "-pack" would choose "pack-metadata,commit-graph" by first
starting from the built-in default and then CLR'ing the "pack"
bit out. If there were already changes made by the lower
precedence configuration files like /etc/gitconfig, the result
might be different and the only definite thing we can say is that
the pack bit is cleared.
- "loose-object" would choose all of the bits by first starting
from the built-in default and then OR'ing the "loose-object" bit
in.
Otherwise, parsing "none" is more or less pointless, as the above
parser always start from 0 and OR's in or CLR's out the named bit.
Whoever writes "none" can just write an empty string, no?
I wonder you'd rather want to do it this way?
parse_fsync_components(var, value, current) {
enum fsync_component positive = 0, negative = 0;
while (string) {
int negated = 0;
enum fsync_component bits;
parse out a single component into <negated, bits>;
if (bits == 0) { /* "none" given */
current = 0;
} else if (negated) {
negative |= bits;
} else {
positive |= bits;
}
advance <string> pointer;
}
return (current | positive) & ~negative;
}
And then ...
> + if (!strcmp(var, "core.fsync")) {
> + if (!value)
> + return config_error_nonbool(var);
> + fsync_components = parse_fsync_components(var, value);
> + return 0;
> + }
> +
... this part would pass the current value of fsync_components as
the third parameter to the parse_fsync_components(). The variable
would be initialized to the FSYNC_COMPONENTS_DEFAULT we saw earlier.
> @@ -1613,7 +1684,7 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
> }
>
> if (!strcmp(var, "core.fsyncobjectfiles")) {
> - fsync_object_files = git_config_bool(var, value);
> + warning(_("core.fsyncobjectfiles is deprecated; use core.fsync instead"));
This is not deprecating but removing the support, which I am not
sure is a sensible thing to do. Rather we should pretend that
core.fsync = "loose-object" (or "-loose-object") were found in the
configuration, shouldn't we?
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v5 1/5] wrapper: move inclusion of CSPRNG headers the wrapper.c file
2022-03-09 23:29 ` Junio C Hamano
@ 2022-03-10 1:21 ` Neeraj Singh
0 siblings, 0 replies; 122+ messages in thread
From: Neeraj Singh @ 2022-03-10 1:21 UTC (permalink / raw)
To: Junio C Hamano
Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker,
Bagas Sanjaya, Elijah Newren,
Ævar Arnfjörð Bjarmason, Patrick Steinhardt,
brian m. carlson, Neeraj K. Singh
On Wed, Mar 9, 2022 at 3:29 PM Junio C Hamano <gitster@pobox.com> wrote:
>
> "Neeraj Singh via GitGitGadget" <gitgitgadget@gmail.com> writes:
>
> > Including NTSecAPI.h in git-compat-util.h causes build errors in any
> > other file that includes winternl.h. That file was included in order to
> > get access to the RtlGenRandom cryptographically secure PRNG. This
> > change scopes the inclusion of all PRNG headers to just the wrapper.c
> > file, which is the only place it is really needed.
>
> It is true that wrapper.c is the only thing that needs these headers
> included as part of its implementation detail of csprng_bytes(), and
> I think I very much like this change for that reason.
>
> Having said that, if it true that including these two header files
> in the same file will lead to compilation failure? That sounds like
> either (1) they represent two mutually incompatible APIs that will
> cause breakage when code that use them, perhaps in two separate
> files to avoid compilation failures, are linked together, or (2)
> these system header files are simply broken. Or something else?
>
> > -/*
> > - * Including the appropriate header file for RtlGenRandom causes MSVC to see a
> > - * redefinition of types in an incompatible way when including headers below.
> > - */
> > -#undef HAVE_RTLGENRANDOM
>
> This comment hints it is more like (1) above? A type used in one
> part of the system is defined differently in other parts of the
> system? I cannot imagine anything but bad things happen when a
> piece of code uses one definition of the type to declare a function,
> and another piece of code uses the other definition of the same type
> to declare a variable and passes it as a parameter to that function.
>
> I do not know this patch makes the situation worse, and I am not a
> Windows person with boxes to dig deeper to begin with. Hence I do
> not mind the change itself, but justifying the change primarily as a
> workaround for some obscure header type clashes on a single system
> leaves a bad taste. If the first sentence weren't there, I wouldn't
> have spent this many minutes wondering if this is a good change ;-)
This is (2), these system header files are simply broken. I've been
looking deeper into why, but haven't bottomed out yet.
Thanks,
Neeraj
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v5 1/5] wrapper: move inclusion of CSPRNG headers the wrapper.c file
2022-03-09 23:03 ` [PATCH v5 1/5] wrapper: move inclusion of CSPRNG headers the wrapper.c file Neeraj Singh via GitGitGadget
2022-03-09 23:29 ` Junio C Hamano
@ 2022-03-10 1:26 ` brian m. carlson
2022-03-10 1:56 ` Neeraj Singh
1 sibling, 1 reply; 122+ messages in thread
From: brian m. carlson @ 2022-03-10 1:26 UTC (permalink / raw)
To: Neeraj Singh via GitGitGadget
Cc: git, rsbecker, bagasdotme, newren, avarab, nksingh85, ps,
Neeraj K. Singh
[-- Attachment #1: Type: text/plain, Size: 1627 bytes --]
On 2022-03-09 at 23:03:14, Neeraj Singh via GitGitGadget wrote:
> From: Neeraj Singh <neerajsi@microsoft.com>
>
> Including NTSecAPI.h in git-compat-util.h causes build errors in any
> other file that includes winternl.h. That file was included in order to
> get access to the RtlGenRandom cryptographically secure PRNG. This
> change scopes the inclusion of all PRNG headers to just the wrapper.c
> file, which is the only place it is really needed.
We generally prefer to do system includes in git-compat-util.h because
it allows us to paper over platform incompatibilities in one place and
to deal with the various ordering problems that can happen on certain
systems.
It may be that Windows needs additional help here; I don't know, because
I don't use Windows. I personally find it unsavoury that Windows ships
with multiple incompatible header files like this, since such problems
are typically avoided by suitable include guards, whose utility has been
well known for several decades. However, if that's the case, let's move
only the Windows changes there, and leave the Unix systems, which lack
this problem, alone.
It would also be helpful to explain the problem that Windows has in more
detail here, including any references to documentation that explains
this incompatibility, so those of us who are not Windows users can more
accurately reason about why we need to be so careful when including
header files there and why this is the best solution (and not, say,
providing our own include guards in a compat file).
--
brian m. carlson (he/him or they/them)
Toronto, Ontario, CA
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v5 1/5] wrapper: move inclusion of CSPRNG headers the wrapper.c file
2022-03-10 1:26 ` brian m. carlson
@ 2022-03-10 1:56 ` Neeraj Singh
0 siblings, 0 replies; 122+ messages in thread
From: Neeraj Singh @ 2022-03-10 1:56 UTC (permalink / raw)
To: brian m. carlson, Neeraj Singh via GitGitGadget, Git List,
Randall S. Becker, Bagas Sanjaya, Elijah Newren,
Ævar Arnfjörð Bjarmason, Neeraj Singh,
Patrick Steinhardt, Neeraj K. Singh
On Wed, Mar 9, 2022 at 5:26 PM brian m. carlson
<sandals@crustytoothpaste.net> wrote:
>
> On 2022-03-09 at 23:03:14, Neeraj Singh via GitGitGadget wrote:
> > From: Neeraj Singh <neerajsi@microsoft.com>
> >
> > Including NTSecAPI.h in git-compat-util.h causes build errors in any
> > other file that includes winternl.h. That file was included in order to
> > get access to the RtlGenRandom cryptographically secure PRNG. This
> > change scopes the inclusion of all PRNG headers to just the wrapper.c
> > file, which is the only place it is really needed.
>
> We generally prefer to do system includes in git-compat-util.h because
> it allows us to paper over platform incompatibilities in one place and
> to deal with the various ordering problems that can happen on certain
> systems.
>
> It may be that Windows needs additional help here; I don't know, because
> I don't use Windows. I personally find it unsavoury that Windows ships
> with multiple incompatible header files like this, since such problems
> are typically avoided by suitable include guards, whose utility has been
> well known for several decades. However, if that's the case, let's move
> only the Windows changes there, and leave the Unix systems, which lack
> this problem, alone.
>
> It would also be helpful to explain the problem that Windows has in more
> detail here, including any references to documentation that explains
> this incompatibility, so those of us who are not Windows users can more
> accurately reason about why we need to be so careful when including
> header files there and why this is the best solution (and not, say,
> providing our own include guards in a compat file).
> --
> brian m. carlson (he/him or they/them)
> Toronto, Ontario, CA
I wasn't able to find any documentation from other people who hit this problem.
The root cause is that NtSecAPI.h has a typedef like this:
```
#ifndef _NTDEF_
typedef LSA_UNICODE_STRING UNICODE_STRING, *PUNICODE_STRING;
typedef LSA_STRING STRING, *PSTRING ;
#endif
```
That's not really appropriate since NtSecAPI.h isn't the correct place
to define this core primitive NT type. It should be including
winternl.h or a similar header.
I'll update the change to only move the Windows definition to the .c file.
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v5 3/5] core.fsync: introduce granular fsync control
2022-03-10 0:21 ` Junio C Hamano
@ 2022-03-10 2:53 ` Neeraj Singh
2022-03-10 7:19 ` Junio C Hamano
` (2 more replies)
0 siblings, 3 replies; 122+ messages in thread
From: Neeraj Singh @ 2022-03-10 2:53 UTC (permalink / raw)
To: Junio C Hamano
Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker,
Bagas Sanjaya, Elijah Newren,
Ævar Arnfjörð Bjarmason, Patrick Steinhardt,
brian m. carlson, Neeraj K. Singh
On Wed, Mar 9, 2022 at 4:21 PM Junio C Hamano <gitster@pobox.com> wrote:
>
> "Neeraj Singh via GitGitGadget" <gitgitgadget@gmail.com> writes:
>
> > +/*
> > + * These values are used to help identify parts of a repository to fsync.
> > + * FSYNC_COMPONENT_NONE identifies data that will not be a persistent part of the
> > + * repository and so shouldn't be fsynced.
> > + */
> > +enum fsync_component {
> > + FSYNC_COMPONENT_NONE,
> > + FSYNC_COMPONENT_LOOSE_OBJECT = 1 << 0,
> > + FSYNC_COMPONENT_PACK = 1 << 1,
> > + FSYNC_COMPONENT_PACK_METADATA = 1 << 2,
> > + FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3,
> > +};
>
> OK, so the idea is that Patrick's "we need to fsync refs" will be
> done by adding a new component to this list, and sprinkling a call
> to fsync_component_or_die() in the code of ref-files backend?
>
Yes. Patrick will need to add fsync_component_or_die wherever his
patch series has already added fsync_or_die.
If he follows Ævar's suggestion of treating remote refs differently
from local refs, he might want to define multiple components.
> I am wondering if fsync_or_die() interface is abstracted well
> enough, or we need things like "the fd is inside this directory; in
> addition to doing the fsync of the fd, please sync the parent
> directory as well" support before we start adding more components
> (if there is such a need, perhaps it comes before this step).
>
I think syncing the parent directory is a separate fsyncMethod that
would require changes across the codebase to obtain an appropriate
directory fd. I'd prefer to treat that as a separable concern.
> > +#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \
> > + FSYNC_COMPONENT_PACK_METADATA | \
> > + FSYNC_COMPONENT_COMMIT_GRAPH)
>
> IOW, everything other than loose object, which already has a
> separate core.fsyncObjectFiles knob to loosen. Everything else we
> currently sync unconditionally and the default keeps that
> arrangement?
>
Yes, trying to keep default behavior identical on non-Windows
platforms. Windows will be expected to adopt batch mode, and have
loose objects in this set.
> > +static inline void fsync_component_or_die(enum fsync_component component, int fd, const char *msg)
> > +{
> > + if (fsync_components & component)
> > + fsync_or_die(fd, msg);
> > +}
>
> Do we have a compelling reason to have this as a static inline
> function? We are talking about concluding an I/O operation and
> I doubt there is a good performance argument for it.
>
Yeah, this is meant to optimize the case where the component isn't
being fsynced. I'll move this function to write-or-die.c below
fsync_or_die.
> > +static const struct fsync_component_entry {
> > + const char *name;
> > + enum fsync_component component_bits;
> > +} fsync_component_table[] = {
>
> thing[] is an array of "thing" (and thing[4] is the "fourth" such
> thing), but this is not an array of a table (it is a name-to-bit
> mapping).
>
> I wonder if this array works without "_table" suffix in its name.
This is modeled after whitespace_rule_names. What if I change this to
the following?
static const struct fsync_component_name {
...
} fsync_component_names[]
>
> > + { "loose-object", FSYNC_COMPONENT_LOOSE_OBJECT },
> > + { "pack", FSYNC_COMPONENT_PACK },
> > + { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA },
> > + { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH },
> > +};
> > +
> > +static enum fsync_component parse_fsync_components(const char *var, const char *string)
> > +{
> > + enum fsync_component output = 0;
> > +
> > + if (!strcmp(string, "none"))
> > + return FSYNC_COMPONENT_NONE;
> > +
> > + while (string) {
> > + int i;
> > + size_t len;
> > + const char *ep;
> > + int negated = 0;
> > + int found = 0;
> > +
> > + string = string + strspn(string, ", \t\n\r");
> > + ep = strchrnul(string, ',');
> > + len = ep - string;
> > +
> > + if (*string == '-') {
> > + negated = 1;
> > + string++;
> > + len--;
> > + if (!len)
> > + warning(_("invalid value for variable %s"), var);
> > + }
> > +
> > + if (!len)
> > + break;
> > +
> > + for (i = 0; i < ARRAY_SIZE(fsync_component_table); ++i) {
> > + const struct fsync_component_entry *entry = &fsync_component_table[i];
> > +
> > + if (strncmp(entry->name, string, len))
> > + continue;
> > +
> > + found = 1;
> > + if (negated)
> > + output &= ~entry->component_bits;
> > + else
> > + output |= entry->component_bits;
> > + }
> > +
> > + if (!found) {
> > + char *component = xstrndup(string, len);
> > + warning(_("ignoring unknown core.fsync component '%s'"), component);
> > + free(component);
> > + }
> > +
> > + string = ep;
> > + }
> > +
> > + return output;
> > +}
>
> Hmph. I would have expected, with built-in default of
> pack,pack-metadata,commit-graph,
>
At the conclusion of this series, I defined 'default' as an aggregate
option that includes
the platform default. I'd prefer not to have any statefulness of the
core.fsync setting so
that there is less confusion about the final fsync configuration. My
colleagues had a fair
amount of confusion internally when testing Git performance internally
with regards to
the core.fsyncObjectFiles setting. Inline this is how your configs
would be written:
> - "none,pack" would choose only "pack" by first clearing the
> built-in default (or whatever was set in configuration files that
> are lower precedence than what we are reading) and then OR'ing
> the "pack" bit in.
>
"pack" would choose only "pack"
> - "-pack" would choose "pack-metadata,commit-graph" by first
> starting from the built-in default and then CLR'ing the "pack"
> bit out. If there were already changes made by the lower
> precedence configuration files like /etc/gitconfig, the result
> might be different and the only definite thing we can say is that
> the pack bit is cleared.
>
"default,-pack" would be the platform default, but not packfiles.
> - "loose-object" would choose all of the bits by first starting
> from the built-in default and then OR'ing the "loose-object" bit
> in.
>
"default,loose-object" would add loose objects to the platform config.
> Otherwise, parsing "none" is more or less pointless, as the above
> parser always start from 0 and OR's in or CLR's out the named bit.
> Whoever writes "none" can just write an empty string, no?
I think the empty string would be disallowed since "core.fsync=" would
be entirely
missing a value. But on testing this doesn't seem to be the case. I'll change
this to be more strict in that the user has to pass an explicit value,
such as 'none'.
>
> I wonder you'd rather want to do it this way?
>
> parse_fsync_components(var, value, current) {
> enum fsync_component positive = 0, negative = 0;
>
> while (string) {
> int negated = 0;
> enum fsync_component bits;
>
> parse out a single component into <negated, bits>;
>
> if (bits == 0) { /* "none" given */
> current = 0;
> } else if (negated) {
> negative |= bits;
> } else {
> positive |= bits;
> }
> advance <string> pointer;
> }
>
> return (current | positive) & ~negative;
> }
>
> And then ...
>
> > + if (!strcmp(var, "core.fsync")) {
> > + if (!value)
> > + return config_error_nonbool(var);
> > + fsync_components = parse_fsync_components(var, value);
> > + return 0;
> > + }
> > +
>
> ... this part would pass the current value of fsync_components as
> the third parameter to the parse_fsync_components(). The variable
> would be initialized to the FSYNC_COMPONENTS_DEFAULT we saw earlier.
>
I'm afraid that this method would lead to statefulness between
different levels configuring
core.fsync. I'd prefer that the user could know what will happen by
just inspecting the value
returned by `git config core.fsync`.
>
> > @@ -1613,7 +1684,7 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
> > }
> >
> > if (!strcmp(var, "core.fsyncobjectfiles")) {
> > - fsync_object_files = git_config_bool(var, value);
> > + warning(_("core.fsyncobjectfiles is deprecated; use core.fsync instead"));
>
> This is not deprecating but removing the support, which I am not
> sure is a sensible thing to do. Rather we should pretend that
> core.fsync = "loose-object" (or "-loose-object") were found in the
> configuration, shouldn't we?
>
The problem I anticipate is that figuring out the final configuration
becomes pretty
complex in the face of conflicting configurations of fsyncObjectFiles
and core.fsync.
The user won't know what will happen without reading the Git code or
doing performance
experiments. I thought we can avoid all of this complexity by just
having a simple warning
that pushes users toward the new configuration value. Aside from
seeing a warning, a
user's actual usage of Git functionality shouldn't be affected.
Alternatively, what if we just silently ignore the old
core.fsyncObjectFiles setting?
If neither of these is an option, I'll put back some support for the
old setting.
Thanks,
Neeraj
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v5 3/5] core.fsync: introduce granular fsync control
2022-03-10 2:53 ` Neeraj Singh
@ 2022-03-10 7:19 ` Junio C Hamano
2022-03-10 18:38 ` Neeraj Singh
2022-03-10 13:11 ` Johannes Schindelin
2022-03-10 17:18 ` Junio C Hamano
2 siblings, 1 reply; 122+ messages in thread
From: Junio C Hamano @ 2022-03-10 7:19 UTC (permalink / raw)
To: Neeraj Singh
Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker,
Bagas Sanjaya, Elijah Newren,
Ævar Arnfjörð Bjarmason, Patrick Steinhardt,
brian m. carlson, Neeraj K. Singh
Neeraj Singh <nksingh85@gmail.com> writes:
> This is modeled after whitespace_rule_names. What if I change this to
> the following?
> static const struct fsync_component_name {
> ...
> } fsync_component_names[]
That's better.
> At the conclusion of this series, I defined 'default' as an aggregate
> option that includes
> the platform default. I'd prefer not to have any statefulness of the
> core.fsync setting so
> that there is less confusion about the final fsync configuration.
Then scratch your preference ;-)
Our configuration files are designed to be "hierarchical" in that
system-wide default can be set in /etc/gitconfig, which can be
overridden by per-user default in $HOME/.gitconfig, which can in
turn be overridden by per-repository setting in .git/config, so
starting from the compiled-in default, reading/augmenting "the value
we tentatively decided based on what we have read so far" with what
we read from lower-precedence files to higher-precedence files is a
norm.
Don't make this little corner of the system different from
everything else; that will only confuse users.
The git_config() callback should expect to see the same var with
different values for that reason. Always restarting from zero will
defeat it.
And always restarting from zero will mean "none" is meaningless,
while it would be a quite nice way to say "forget everything we have
read so far and start from scratch" when you really want to refuse
what the system default wants to give you.
>> > @@ -1613,7 +1684,7 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
>> > }
>> >
>> > if (!strcmp(var, "core.fsyncobjectfiles")) {
>> > - fsync_object_files = git_config_bool(var, value);
>> > + warning(_("core.fsyncobjectfiles is deprecated; use core.fsync instead"));
>>
>> This is not deprecating but removing the support, which I am not
>> sure is a sensible thing to do. Rather we should pretend that
>> core.fsync = "loose-object" (or "-loose-object") were found in the
>> configuration, shouldn't we?
>>
>
> The problem I anticipate is that figuring out the final configuration
> becomes pretty
> complex in the face of conflicting configurations of fsyncObjectFiles
> and core.fsync.
Don't start your thinking from too complex configuration that mixes
and matches. Instead, think what happens to folks who are *NOT*
interested in the new way of doing this. They aren't interested in
setting core.fsync, and they already have core.fsyncObjectFiles set.
You want to make sure their experience does not suck.
The quoted code simply _IGNORES_ their wish and forces whatever
default configuration core.fsync codepath happens to choose, which
is a grave regression from their point of view.
One way to handle this more gracefully is to delay the final
decision until the end of the configuraiton file processing, and
keep track of core.fsyncObjectFiles and core.fsync separately. If
the latter is never set but the former is, then you are dealing with
such a user who hasn't migrated. Give them a warning (the text
above is fine---we can tell them "that's deprecated and you should
use the other one instead"), but in the meantime, until deprecation
turns into removal of support, keep honoring their original wish.
If core.fsync is set to something, you can still give them a warning
when you see core.fsyncObjectFiles, saying "that's deprecated and
because you have core.fsync, we'll ignore the old one", and use the
new method exclusively, without having to worry about mixing.
^ permalink raw reply [flat|nested] 122+ messages in thread
* Future-proofed syncing of refs
2022-03-09 23:03 ` [PATCH v5 0/5] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget
` (4 preceding siblings ...)
2022-03-09 23:03 ` [PATCH v5 5/5] core.fsync: documentation and user-friendly aggregate options Neeraj Singh via GitGitGadget
@ 2022-03-10 9:53 ` Patrick Steinhardt
2022-03-10 9:53 ` [PATCH 6/8] core.fsync: add `fsync_component()` wrapper which doesn't die Patrick Steinhardt
` (3 subsequent siblings)
9 siblings, 0 replies; 122+ messages in thread
From: Patrick Steinhardt @ 2022-03-10 9:53 UTC (permalink / raw)
To: git
Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, sandals,
Neeraj K. Singh, Junio C Hamano
[-- Attachment #1: Type: text/plain, Size: 1472 bytes --]
Hi,
these three patches apply on top of Neeraj's v5 of his "A design for
future-proofing fsync() configuration". I'm sending this as a reply to
his v5 to keep the discussion in one place -- I think ultimately, we may
want to merge both series into a single one anyway. But please shout at
me if this is considered bad style and I'll split it out into a separate
thread.
In any case, these three patches implement fsyncing for loose and packed
references using the proposed `core.fsync` option, with three additional
knobs:
- "loose-ref" will fsync loose references.
- "packed-refs" will fsync packed references.
- "refs" will fsync all references, which should ideally also
include all new backends like the reftable backend.
I think this extension demonstrates that the design proposed by Neeraj
is quite easy to extend without too much boilerplate.
Patrick
Patrick Steinhardt (3):
core.fsync: add `fsync_component()` wrapper which doesn't die
core.fsync: new option to harden loose references
core.fsync: new option to harden packed references
Documentation/config/core.txt | 3 +++
cache.h | 22 ++++++++++++++++++----
config.c | 3 +++
refs/files-backend.c | 29 +++++++++++++++++++++++++++++
refs/packed-backend.c | 3 ++-
write-or-die.c | 10 ++++++----
6 files changed, 61 insertions(+), 9 deletions(-)
--
2.35.1
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 122+ messages in thread
* [PATCH 6/8] core.fsync: add `fsync_component()` wrapper which doesn't die
2022-03-09 23:03 ` [PATCH v5 0/5] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget
` (5 preceding siblings ...)
2022-03-10 9:53 ` Future-proofed syncing of refs Patrick Steinhardt
@ 2022-03-10 9:53 ` Patrick Steinhardt
2022-03-10 17:34 ` Junio C Hamano
2022-03-10 9:53 ` [PATCH 7/8] core.fsync: new option to harden loose references Patrick Steinhardt
` (2 subsequent siblings)
9 siblings, 1 reply; 122+ messages in thread
From: Patrick Steinhardt @ 2022-03-10 9:53 UTC (permalink / raw)
To: git
Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, sandals,
Neeraj K. Singh, Junio C Hamano
[-- Attachment #1: Type: text/plain, Size: 2142 bytes --]
We have a `fsync_component_or_die()` helper function which only syncs
changes to disk in case the corresponding config is enabled by the user.
This wrapper will always die on an error though, which makes it
insufficient for new callsites we are about to add.
Add a new `fsync_component()` wrapper which returns an error code
instead of dying.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
cache.h | 13 ++++++++++---
write-or-die.c | 10 ++++++----
2 files changed, 16 insertions(+), 7 deletions(-)
diff --git a/cache.h b/cache.h
index f307e89516..63a95d1977 100644
--- a/cache.h
+++ b/cache.h
@@ -1745,12 +1745,19 @@ int copy_file(const char *dst, const char *src, int mode);
int copy_file_with_time(const char *dst, const char *src, int mode);
void write_or_die(int fd, const void *buf, size_t count);
-void fsync_or_die(int fd, const char *);
+int maybe_fsync(int fd);
+
+static inline int fsync_component(enum fsync_component component, int fd)
+{
+ if (fsync_components & component)
+ return maybe_fsync(fd);
+ return 0;
+}
static inline void fsync_component_or_die(enum fsync_component component, int fd, const char *msg)
{
- if (fsync_components & component)
- fsync_or_die(fd, msg);
+ if (fsync_component(component, fd) < 0)
+ die_errno("fsync error on '%s'", msg);
}
ssize_t read_in_full(int fd, void *buf, size_t count);
diff --git a/write-or-die.c b/write-or-die.c
index 9faa5f9f56..4a5455ce46 100644
--- a/write-or-die.c
+++ b/write-or-die.c
@@ -56,19 +56,21 @@ void fprintf_or_die(FILE *f, const char *fmt, ...)
}
}
-void fsync_or_die(int fd, const char *msg)
+int maybe_fsync(int fd)
{
if (use_fsync < 0)
use_fsync = git_env_bool("GIT_TEST_FSYNC", 1);
if (!use_fsync)
- return;
+ return 0;
if (fsync_method == FSYNC_METHOD_WRITEOUT_ONLY &&
git_fsync(fd, FSYNC_WRITEOUT_ONLY) >= 0)
- return;
+ return 0;
if (git_fsync(fd, FSYNC_HARDWARE_FLUSH) < 0)
- die_errno("fsync error on '%s'", msg);
+ return -1;
+
+ return 0;
}
void write_or_die(int fd, const void *buf, size_t count)
--
2.35.1
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH 7/8] core.fsync: new option to harden loose references
2022-03-09 23:03 ` [PATCH v5 0/5] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget
` (6 preceding siblings ...)
2022-03-10 9:53 ` [PATCH 6/8] core.fsync: add `fsync_component()` wrapper which doesn't die Patrick Steinhardt
@ 2022-03-10 9:53 ` Patrick Steinhardt
2022-03-10 18:25 ` Junio C Hamano
` (2 more replies)
2022-03-10 9:53 ` [PATCH 8/8] core.fsync: new option to harden packed references Patrick Steinhardt
2022-03-10 22:43 ` [PATCH v6 0/6] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget
9 siblings, 3 replies; 122+ messages in thread
From: Patrick Steinhardt @ 2022-03-10 9:53 UTC (permalink / raw)
To: git
Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, sandals,
Neeraj K. Singh, Junio C Hamano
[-- Attachment #1: Type: text/plain, Size: 6283 bytes --]
When writing loose references to disk we first create a lockfile, write
the updated value of the reference into that lockfile, and on commit we
rename the file into place. According to filesystem developers, this
behaviour is broken because applications should always sync data to disk
before doing the final rename to ensure data consistency [1][2][3]. If
applications fail to do this correctly, a hard crash of the machine can
easily result in corrupted on-disk data.
This kind of corruption can in fact be easily observed with Git when the
machine hard-crashes shortly after writing loose references to disk. On
machines with ext4, this will likely lead to the "empty files" problem:
the file has been renamed, but its data has not been synced to disk. The
result is that the references is corrupt, and in the worst case this can
lead to data loss.
Implement a new option to harden loose references so that users and
admins can avoid this scenario by syncing locked loose references to
disk before we rename them into place.
[1]: https://thunk.org/tytso/blog/2009/03/15/dont-fear-the-fsync/
[2]: https://btrfs.wiki.kernel.org/index.php/FAQ (What are the crash guarantees of overwrite-by-rename)
[3]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/admin-guide/ext4.rst (see auto_da_alloc)
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
Documentation/config/core.txt | 2 ++
cache.h | 6 +++++-
config.c | 2 ++
refs/files-backend.c | 29 +++++++++++++++++++++++++++++
4 files changed, 38 insertions(+), 1 deletion(-)
diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
index 973805e8a9..b67d3c340e 100644
--- a/Documentation/config/core.txt
+++ b/Documentation/config/core.txt
@@ -564,8 +564,10 @@ core.fsync::
* `pack-metadata` hardens packfile bitmaps and indexes.
* `commit-graph` hardens the commit graph file.
* `index` hardens the index when it is modified.
+* `loose-ref` hardens references modified in the repo in loose-ref form.
* `objects` is an aggregate option that is equivalent to
`loose-object,pack`.
+* `refs` is an aggregate option that is equivalent to `loose-ref`.
* `derived-metadata` is an aggregate option that is equivalent to
`pack-metadata,commit-graph`.
* `default` is an aggregate option that is equivalent to
diff --git a/cache.h b/cache.h
index 63a95d1977..b56a56f539 100644
--- a/cache.h
+++ b/cache.h
@@ -1005,11 +1005,14 @@ enum fsync_component {
FSYNC_COMPONENT_PACK_METADATA = 1 << 2,
FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3,
FSYNC_COMPONENT_INDEX = 1 << 4,
+ FSYNC_COMPONENT_LOOSE_REF = 1 << 5,
};
#define FSYNC_COMPONENTS_OBJECTS (FSYNC_COMPONENT_LOOSE_OBJECT | \
FSYNC_COMPONENT_PACK)
+#define FSYNC_COMPONENTS_REFS (FSYNC_COMPONENT_LOOSE_REF)
+
#define FSYNC_COMPONENTS_DERIVED_METADATA (FSYNC_COMPONENT_PACK_METADATA | \
FSYNC_COMPONENT_COMMIT_GRAPH)
@@ -1026,7 +1029,8 @@ enum fsync_component {
FSYNC_COMPONENT_PACK | \
FSYNC_COMPONENT_PACK_METADATA | \
FSYNC_COMPONENT_COMMIT_GRAPH | \
- FSYNC_COMPONENT_INDEX)
+ FSYNC_COMPONENT_INDEX | \
+ FSYNC_COMPONENT_LOOSE_REF)
/*
* A bitmask indicating which components of the repo should be fsynced.
diff --git a/config.c b/config.c
index f03f27c3de..b5d3e6e404 100644
--- a/config.c
+++ b/config.c
@@ -1332,7 +1332,9 @@ static const struct fsync_component_entry {
{ "pack-metadata", FSYNC_COMPONENT_PACK_METADATA },
{ "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH },
{ "index", FSYNC_COMPONENT_INDEX },
+ { "loose-ref", FSYNC_COMPONENT_LOOSE_REF },
{ "objects", FSYNC_COMPONENTS_OBJECTS },
+ { "refs", FSYNC_COMPONENTS_REFS },
{ "derived-metadata", FSYNC_COMPONENTS_DERIVED_METADATA },
{ "default", FSYNC_COMPONENTS_DEFAULT },
{ "committed", FSYNC_COMPONENTS_COMMITTED },
diff --git a/refs/files-backend.c b/refs/files-backend.c
index f59589d6cc..279316de45 100644
--- a/refs/files-backend.c
+++ b/refs/files-backend.c
@@ -1392,6 +1392,15 @@ static int refs_rename_ref_available(struct ref_store *refs,
return ok;
}
+static int files_sync_loose_ref(struct ref_lock *lock, struct strbuf *err)
+{
+ int ret = fsync_component(FSYNC_COMPONENT_LOOSE_REF, get_lock_file_fd(&lock->lk));
+ if (ret)
+ strbuf_addf(err, "could not sync loose ref '%s': %s", lock->ref_name,
+ strerror(errno));
+ return ret;
+}
+
static int files_copy_or_rename_ref(struct ref_store *ref_store,
const char *oldrefname, const char *newrefname,
const char *logmsg, int copy)
@@ -1504,6 +1513,7 @@ static int files_copy_or_rename_ref(struct ref_store *ref_store,
oidcpy(&lock->old_oid, &orig_oid);
if (write_ref_to_lockfile(lock, &orig_oid, 0, &err) ||
+ files_sync_loose_ref(lock, &err) ||
commit_ref_update(refs, lock, &orig_oid, logmsg, &err)) {
error("unable to write current sha1 into %s: %s", newrefname, err.buf);
strbuf_release(&err);
@@ -1524,6 +1534,7 @@ static int files_copy_or_rename_ref(struct ref_store *ref_store,
flag = log_all_ref_updates;
log_all_ref_updates = LOG_REFS_NONE;
if (write_ref_to_lockfile(lock, &orig_oid, 0, &err) ||
+ files_sync_loose_ref(lock, &err) ||
commit_ref_update(refs, lock, &orig_oid, NULL, &err)) {
error("unable to write current sha1 into %s: %s", oldrefname, err.buf);
strbuf_release(&err);
@@ -2819,6 +2830,24 @@ static int files_transaction_prepare(struct ref_store *ref_store,
}
}
+ /*
+ * Sync all lockfiles to disk to ensure data consistency. We do this in
+ * a separate step such that we can sync all modified refs in a single
+ * step, which may be more efficient on some filesystems.
+ */
+ for (i = 0; i < transaction->nr; i++) {
+ struct ref_update *update = transaction->updates[i];
+ struct ref_lock *lock = update->backend_data;
+
+ if (!(update->flags & REF_NEEDS_COMMIT))
+ continue;
+
+ if (files_sync_loose_ref(lock, err)) {
+ ret = TRANSACTION_GENERIC_ERROR;
+ goto cleanup;
+ }
+ }
+
cleanup:
free(head_ref);
string_list_clear(&affected_refnames, 0);
--
2.35.1
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH 8/8] core.fsync: new option to harden packed references
2022-03-09 23:03 ` [PATCH v5 0/5] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget
` (7 preceding siblings ...)
2022-03-10 9:53 ` [PATCH 7/8] core.fsync: new option to harden loose references Patrick Steinhardt
@ 2022-03-10 9:53 ` Patrick Steinhardt
2022-03-10 18:28 ` Junio C Hamano
2022-03-10 22:43 ` [PATCH v6 0/6] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget
9 siblings, 1 reply; 122+ messages in thread
From: Patrick Steinhardt @ 2022-03-10 9:53 UTC (permalink / raw)
To: git
Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, sandals,
Neeraj K. Singh, Junio C Hamano
[-- Attachment #1: Type: text/plain, Size: 3485 bytes --]
Similar to the preceding commit, this commit adds a new option to harden
packed references so that users and admins can avoid data loss when we
commit a new packed-refs file.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
Documentation/config/core.txt | 3 ++-
cache.h | 7 +++++--
config.c | 1 +
refs/packed-backend.c | 3 ++-
4 files changed, 10 insertions(+), 4 deletions(-)
diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
index b67d3c340e..3fd466f955 100644
--- a/Documentation/config/core.txt
+++ b/Documentation/config/core.txt
@@ -565,9 +565,10 @@ core.fsync::
* `commit-graph` hardens the commit graph file.
* `index` hardens the index when it is modified.
* `loose-ref` hardens references modified in the repo in loose-ref form.
+* `packed-refs` hardens references modified in the repo in packed-refs form.
* `objects` is an aggregate option that is equivalent to
`loose-object,pack`.
-* `refs` is an aggregate option that is equivalent to `loose-ref`.
+* `refs` is an aggregate option that is equivalent to `loose-ref,packed-refs`.
* `derived-metadata` is an aggregate option that is equivalent to
`pack-metadata,commit-graph`.
* `default` is an aggregate option that is equivalent to
diff --git a/cache.h b/cache.h
index b56a56f539..9b7c282fa5 100644
--- a/cache.h
+++ b/cache.h
@@ -1006,12 +1006,14 @@ enum fsync_component {
FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3,
FSYNC_COMPONENT_INDEX = 1 << 4,
FSYNC_COMPONENT_LOOSE_REF = 1 << 5,
+ FSYNC_COMPONENT_PACKED_REFS = 1 << 6,
};
#define FSYNC_COMPONENTS_OBJECTS (FSYNC_COMPONENT_LOOSE_OBJECT | \
FSYNC_COMPONENT_PACK)
-#define FSYNC_COMPONENTS_REFS (FSYNC_COMPONENT_LOOSE_REF)
+#define FSYNC_COMPONENTS_REFS (FSYNC_COMPONENT_LOOSE_REF | \
+ FSYNC_COMPONENT_PACKED_REFS)
#define FSYNC_COMPONENTS_DERIVED_METADATA (FSYNC_COMPONENT_PACK_METADATA | \
FSYNC_COMPONENT_COMMIT_GRAPH)
@@ -1030,7 +1032,8 @@ enum fsync_component {
FSYNC_COMPONENT_PACK_METADATA | \
FSYNC_COMPONENT_COMMIT_GRAPH | \
FSYNC_COMPONENT_INDEX | \
- FSYNC_COMPONENT_LOOSE_REF)
+ FSYNC_COMPONENT_LOOSE_REF | \
+ FSYNC_COMPONENT_PACKED_REFS)
/*
* A bitmask indicating which components of the repo should be fsynced.
diff --git a/config.c b/config.c
index b5d3e6e404..b4a2ee3a8c 100644
--- a/config.c
+++ b/config.c
@@ -1333,6 +1333,7 @@ static const struct fsync_component_entry {
{ "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH },
{ "index", FSYNC_COMPONENT_INDEX },
{ "loose-ref", FSYNC_COMPONENT_LOOSE_REF },
+ { "packed-refs", FSYNC_COMPONENT_PACKED_REFS },
{ "objects", FSYNC_COMPONENTS_OBJECTS },
{ "refs", FSYNC_COMPONENTS_REFS },
{ "derived-metadata", FSYNC_COMPONENTS_DERIVED_METADATA },
diff --git a/refs/packed-backend.c b/refs/packed-backend.c
index 27dd8c3922..32d6635969 100644
--- a/refs/packed-backend.c
+++ b/refs/packed-backend.c
@@ -1262,7 +1262,8 @@ static int write_with_updates(struct packed_ref_store *refs,
goto error;
}
- if (close_tempfile_gently(refs->tempfile)) {
+ if (fsync_component(FSYNC_COMPONENT_PACKED_REFS, get_tempfile_fd(refs->tempfile)) ||
+ close_tempfile_gently(refs->tempfile)) {
strbuf_addf(err, "error closing file %s: %s",
get_tempfile_path(refs->tempfile),
strerror(errno));
--
2.35.1
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply related [flat|nested] 122+ messages in thread
* Re: [PATCH v4 2/4] core.fsync: introduce granular fsync control
2022-03-09 20:03 ` Junio C Hamano
@ 2022-03-10 12:33 ` Patrick Steinhardt
2022-03-10 17:15 ` Junio C Hamano
0 siblings, 1 reply; 122+ messages in thread
From: Patrick Steinhardt @ 2022-03-10 12:33 UTC (permalink / raw)
To: Junio C Hamano
Cc: Neeraj Singh, Neeraj Singh via GitGitGadget, Git List,
Randall S. Becker, Bagas Sanjaya, Elijah Newren,
Ævar Arnfjörð Bjarmason, Neeraj K. Singh
[-- Attachment #1: Type: text/plain, Size: 1930 bytes --]
On Wed, Mar 09, 2022 at 12:03:11PM -0800, Junio C Hamano wrote:
> Patrick Steinhardt <ps@pks.im> writes:
>
> >> If the user cares about fsynching loose object files in the right
> >> way, we shouldn't leave loose ref files not following the safe
> >> safety level, regardless of how this new core.fsync knobs would look
> >> like.
> >>
> >> I think we three are in agreement on that.
> >
> > Is there anything I can specifically do to help out with this topic? We
> > have again hit data loss in production because we don't sync loose refs
> > to disk before renaming them into place, so I'd really love to sort out
> > this issue somehow so that I can revive my patch series which fixes the
> > known repository corruption [1].
>
> How about doing a series to unconditionally sync loose ref creation
> and modification?
>
> Alternatively, we could link it to the existing configuration to
> control synching of object files.
>
> I do not think core.fsyncObjectFiles having "object" in its name is
> a good reason not to think those who set it to true only care about
> the loose object files and nothing else. It is more sensible to
> consider that those who set it to true cares about the repository
> integrity more than those who set it to false, I would think.
>
> But that (i.e. doing it conditionally and choose which knob to use)
> is one extra thing that needs justification, so starting from
> unconditional fsync_or_die() may be the best way to ease it in.
I'd be happy to revive my old patch series, but this kind of depends on
where we're standing with the other patch series by Neeraj. If you say
that we'll likely not land his patch series for the upcoming release,
but a small patch series which only starts to sync loose refs may have a
chance, then I'd like to go down that path as a stop-gap solution.
Otherwise it probably wouldn't make a lot of sense.
Patrick
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v5 3/5] core.fsync: introduce granular fsync control
2022-03-10 2:53 ` Neeraj Singh
2022-03-10 7:19 ` Junio C Hamano
@ 2022-03-10 13:11 ` Johannes Schindelin
2022-03-10 17:18 ` Junio C Hamano
2 siblings, 0 replies; 122+ messages in thread
From: Johannes Schindelin @ 2022-03-10 13:11 UTC (permalink / raw)
To: Neeraj Singh
Cc: Junio C Hamano, Neeraj Singh via GitGitGadget, Git List,
Randall S. Becker, Bagas Sanjaya, Elijah Newren,
Ævar Arnfjörð Bjarmason, Patrick Steinhardt,
brian m. carlson, Neeraj K. Singh
Hi Neeraj,
On Wed, 9 Mar 2022, Neeraj Singh wrote:
> On Wed, Mar 9, 2022 at 4:21 PM Junio C Hamano <gitster@pobox.com> wrote:
>
> > I am wondering if fsync_or_die() interface is abstracted well enough,
> > or we need things like "the fd is inside this directory; in addition
> > to doing the fsync of the fd, please sync the parent directory as
> > well" support before we start adding more components (if there is such
> > a need, perhaps it comes before this step).
> >
>
> I think syncing the parent directory is a separate fsyncMethod that
> would require changes across the codebase to obtain an appropriate
> directory fd. I'd prefer to treat that as a separable concern.
That makes sense to me because I expect further abstraction to be
necessary here because Unix/Linux semantics differ quite a bit more from
Windows semantics when it comes to directory "file" descriptors than when
talking about files' file descriptors.
> > > +#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \
> > > + FSYNC_COMPONENT_PACK_METADATA | \
> > > + FSYNC_COMPONENT_COMMIT_GRAPH)
> >
> > IOW, everything other than loose object, which already has a
> > separate core.fsyncObjectFiles knob to loosen. Everything else we
> > currently sync unconditionally and the default keeps that
> > arrangement?
> >
>
> Yes, trying to keep default behavior identical on non-Windows
> platforms. Windows will be expected to adopt batch mode, and have
> loose objects in this set.
We already adopted an early version of this patch series:
https://github.com/git-for-windows/git/commit/98209a5f6e4
And yes, I will gladly adapt that to whatever lands in core Git.
Thank you _so_ much for working on this!
Dscho
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v4 2/4] core.fsync: introduce granular fsync control
2022-03-10 12:33 ` Patrick Steinhardt
@ 2022-03-10 17:15 ` Junio C Hamano
0 siblings, 0 replies; 122+ messages in thread
From: Junio C Hamano @ 2022-03-10 17:15 UTC (permalink / raw)
To: Patrick Steinhardt
Cc: Neeraj Singh, Neeraj Singh via GitGitGadget, Git List,
Randall S. Becker, Bagas Sanjaya, Elijah Newren,
Ævar Arnfjörð Bjarmason, Neeraj K. Singh
Patrick Steinhardt <ps@pks.im> writes:
>> How about doing a series to unconditionally sync loose ref creation
>> and modification?
>>
>> Alternatively, we could link it to the existing configuration to
>> control synching of object files.
>>
>> I do not think core.fsyncObjectFiles having "object" in its name is
>> a good reason not to think those who set it to true only care about
>> the loose object files and nothing else. It is more sensible to
>> consider that those who set it to true cares about the repository
>> integrity more than those who set it to false, I would think.
>>
>> But that (i.e. doing it conditionally and choose which knob to use)
>> is one extra thing that needs justification, so starting from
>> unconditional fsync_or_die() may be the best way to ease it in.
>
> I'd be happy to revive my old patch series, but this kind of depends on
> where we're standing with the other patch series by Neeraj. If you say
> that we'll likely not land his patch series for the upcoming release,
> but a small patch series which only starts to sync loose refs may have a
> chance, then I'd like to go down that path as a stop-gap solution.
> Otherwise it probably wouldn't make a lot of sense.
The above was what I wrote before the revived series from Neeraj.
Now I've seen it, and more importantly, the most recent one from you
on top of that to add ref hardening as a new "component" or two, I
like the overall shape of the end result (except for the semantics
implemented by the configuration parser, which can be fixed without
affecting how the hardening components are implemented). Hopefully
the base series of Neeraj can be solidified soon enough to make it
unnecessary for a stop-gap measure. We'll see.
Thanks.
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v5 3/5] core.fsync: introduce granular fsync control
2022-03-10 2:53 ` Neeraj Singh
2022-03-10 7:19 ` Junio C Hamano
2022-03-10 13:11 ` Johannes Schindelin
@ 2022-03-10 17:18 ` Junio C Hamano
2 siblings, 0 replies; 122+ messages in thread
From: Junio C Hamano @ 2022-03-10 17:18 UTC (permalink / raw)
To: Neeraj Singh
Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker,
Bagas Sanjaya, Elijah Newren,
Ævar Arnfjörð Bjarmason, Patrick Steinhardt,
brian m. carlson, Neeraj K. Singh
Neeraj Singh <nksingh85@gmail.com> writes:
>> I am wondering if fsync_or_die() interface is abstracted well
>> enough, or we need things like "the fd is inside this directory; in
>> addition to doing the fsync of the fd, please sync the parent
>> directory as well" support before we start adding more components
>> (if there is such a need, perhaps it comes before this step).
>>
>
> I think syncing the parent directory is a separate fsyncMethod that
> would require changes across the codebase to obtain an appropriate
> directory fd. I'd prefer to treat that as a separable concern.
Yeah, that would be a sensible direction to go. If we never did the
"sync the parent" thing, we do not need it in the fsyncMethod world
immediately. It can be added later.
Thanks.
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH 6/8] core.fsync: add `fsync_component()` wrapper which doesn't die
2022-03-10 9:53 ` [PATCH 6/8] core.fsync: add `fsync_component()` wrapper which doesn't die Patrick Steinhardt
@ 2022-03-10 17:34 ` Junio C Hamano
2022-03-10 18:40 ` Neeraj Singh
0 siblings, 1 reply; 122+ messages in thread
From: Junio C Hamano @ 2022-03-10 17:34 UTC (permalink / raw)
To: Patrick Steinhardt
Cc: git, rsbecker, bagasdotme, newren, avarab, nksingh85, sandals,
Neeraj K. Singh
Patrick Steinhardt <ps@pks.im> writes:
> We have a `fsync_component_or_die()` helper function which only syncs
> changes to disk in case the corresponding config is enabled by the user.
> This wrapper will always die on an error though, which makes it
> insufficient for new callsites we are about to add.
You can replace "which makes it ..." part with a bit more concrete
description to save suspense from the readers.
fsync_component_or_die() that dies upon an error is not useful
for callers with their own error handling or recovery logic,
like ref transaction API.
Split fsync_component() out that returns an error to help them.
> -void fsync_or_die(int fd, const char *);
> +int maybe_fsync(int fd);
> ...
> +static inline int fsync_component(enum fsync_component component, int fd)
> +{
> + if (fsync_components & component)
> + return maybe_fsync(fd);
> + return 0;
> +}
>
> static inline void fsync_component_or_die(enum fsync_component component, int fd, const char *msg)
> {
> - if (fsync_components & component)
> - fsync_or_die(fd, msg);
> + if (fsync_component(component, fd) < 0)
> + die_errno("fsync error on '%s'", msg);
> }
I think in the eventuall reroll, these "static inline" functions on
the I/O code path will become real functions in write-or-die.c but
other than that this reorganization looks sensible.
Thanks.
> diff --git a/write-or-die.c b/write-or-die.c
> index 9faa5f9f56..4a5455ce46 100644
> --- a/write-or-die.c
> +++ b/write-or-die.c
> @@ -56,19 +56,21 @@ void fprintf_or_die(FILE *f, const char *fmt, ...)
> }
> }
>
> -void fsync_or_die(int fd, const char *msg)
> +int maybe_fsync(int fd)
> {
> if (use_fsync < 0)
> use_fsync = git_env_bool("GIT_TEST_FSYNC", 1);
> if (!use_fsync)
> - return;
> + return 0;
>
> if (fsync_method == FSYNC_METHOD_WRITEOUT_ONLY &&
> git_fsync(fd, FSYNC_WRITEOUT_ONLY) >= 0)
> - return;
> + return 0;
>
> if (git_fsync(fd, FSYNC_HARDWARE_FLUSH) < 0)
> - die_errno("fsync error on '%s'", msg);
> + return -1;
> +
> + return 0;
> }
>
> void write_or_die(int fd, const void *buf, size_t count)
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH 7/8] core.fsync: new option to harden loose references
2022-03-10 9:53 ` [PATCH 7/8] core.fsync: new option to harden loose references Patrick Steinhardt
@ 2022-03-10 18:25 ` Junio C Hamano
2022-03-10 19:03 ` Neeraj Singh
2022-03-10 22:54 ` Neeraj Singh
2022-03-11 6:40 ` Junio C Hamano
2 siblings, 1 reply; 122+ messages in thread
From: Junio C Hamano @ 2022-03-10 18:25 UTC (permalink / raw)
To: Patrick Steinhardt
Cc: git, rsbecker, bagasdotme, newren, avarab, nksingh85, sandals,
Neeraj K. Singh
Patrick Steinhardt <ps@pks.im> writes:
> diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
> index 973805e8a9..b67d3c340e 100644
> --- a/Documentation/config/core.txt
> +++ b/Documentation/config/core.txt
> @@ -564,8 +564,10 @@ core.fsync::
> * `pack-metadata` hardens packfile bitmaps and indexes.
> * `commit-graph` hardens the commit graph file.
> * `index` hardens the index when it is modified.
> +* `loose-ref` hardens references modified in the repo in loose-ref form.
> * `objects` is an aggregate option that is equivalent to
> `loose-object,pack`.
> +* `refs` is an aggregate option that is equivalent to `loose-ref`.
Aggregate of one feels strange. I do not see a strong reason to
have two separate classes loose vs packed and allow them be
protected to different robustness, but that aside, if we were to
have two, this "aggregate" is better added when the second one is.
Having said that, given that a separate ref backend that has no
distinction between loose or packed is on horizen, I think we would
rather prefer to see a single "ref" component that governs all
backends.
> @@ -1026,7 +1029,8 @@ enum fsync_component {
> FSYNC_COMPONENT_PACK | \
> FSYNC_COMPONENT_PACK_METADATA | \
> FSYNC_COMPONENT_COMMIT_GRAPH | \
> - FSYNC_COMPONENT_INDEX)
> + FSYNC_COMPONENT_INDEX | \
> + FSYNC_COMPONENT_LOOSE_REF)
OK.
> +static int files_sync_loose_ref(struct ref_lock *lock, struct strbuf *err)
This file-scope static function will not be in the vtable (in other
words, it is not like "sync_loose_ref" method must be defined across
all ref backends and this function is called as the implementation
of the method for the files backend), so we do not have to give it
"files_" prefix if we do not want to.
sync_loose_ref() may be easier to read, perhaps?
> +{
> + int ret = fsync_component(FSYNC_COMPONENT_LOOSE_REF, get_lock_file_fd(&lock->lk));
> + if (ret)
> + strbuf_addf(err, "could not sync loose ref '%s': %s", lock->ref_name,
> + strerror(errno));
> + return ret;
> +}
OK.
Good illustration how the new helper in 6/8 is useful. It would be
nice if the reroll of the base topic by Neeraj reorders the patches
to introduce fsync_component() much earlier, at the same time it
introduces the fsync_component_or_die().
> @@ -1504,6 +1513,7 @@ static int files_copy_or_rename_ref(struct ref_store *ref_store,
> oidcpy(&lock->old_oid, &orig_oid);
>
> if (write_ref_to_lockfile(lock, &orig_oid, 0, &err) ||
> + files_sync_loose_ref(lock, &err) ||
> commit_ref_update(refs, lock, &orig_oid, logmsg, &err)) {
> error("unable to write current sha1 into %s: %s", newrefname, err.buf);
> strbuf_release(&err);
> @@ -1524,6 +1534,7 @@ static int files_copy_or_rename_ref(struct ref_store *ref_store,
> flag = log_all_ref_updates;
> log_all_ref_updates = LOG_REFS_NONE;
> if (write_ref_to_lockfile(lock, &orig_oid, 0, &err) ||
> + files_sync_loose_ref(lock, &err) ||
> commit_ref_update(refs, lock, &orig_oid, NULL, &err)) {
> error("unable to write current sha1 into %s: %s", oldrefname, err.buf);
> strbuf_release(&err);
We used to skip commit_ref_update() and gone to the error code path
upon write failure, and we do the same upon fsync failure now. The
error code path will do the rollback the same way as before.
OK, these look sensible.
> @@ -2819,6 +2830,24 @@ static int files_transaction_prepare(struct ref_store *ref_store,
> }
> }
>
> + /*
> + * Sync all lockfiles to disk to ensure data consistency. We do this in
> + * a separate step such that we can sync all modified refs in a single
> + * step, which may be more efficient on some filesystems.
> + */
> + for (i = 0; i < transaction->nr; i++) {
> + struct ref_update *update = transaction->updates[i];
> + struct ref_lock *lock = update->backend_data;
> +
> + if (!(update->flags & REF_NEEDS_COMMIT))
> + continue;
> +
> + if (files_sync_loose_ref(lock, err)) {
> + ret = TRANSACTION_GENERIC_ERROR;
> + goto cleanup;
> + }
> + }
An obvious alternative that naïvely comes to mind is to keep going
after seeing the first failure to sync and attempt to sync all the
rest, but remember that we had an error. But I think what this
patch does makes a lot more sense, as the error code path will just
cleans the transaction up. If any of them fails, there is no reason
to spend more effort.
Thanks.
> cleanup:
> free(head_ref);
> string_list_clear(&affected_refnames, 0);
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH 8/8] core.fsync: new option to harden packed references
2022-03-10 9:53 ` [PATCH 8/8] core.fsync: new option to harden packed references Patrick Steinhardt
@ 2022-03-10 18:28 ` Junio C Hamano
2022-03-11 9:10 ` Patrick Steinhardt
0 siblings, 1 reply; 122+ messages in thread
From: Junio C Hamano @ 2022-03-10 18:28 UTC (permalink / raw)
To: Patrick Steinhardt
Cc: git, rsbecker, bagasdotme, newren, avarab, nksingh85, sandals,
Neeraj K. Singh
Patrick Steinhardt <ps@pks.im> writes:
> diff --git a/refs/packed-backend.c b/refs/packed-backend.c
> index 27dd8c3922..32d6635969 100644
> --- a/refs/packed-backend.c
> +++ b/refs/packed-backend.c
> @@ -1262,7 +1262,8 @@ static int write_with_updates(struct packed_ref_store *refs,
> goto error;
> }
>
> - if (close_tempfile_gently(refs->tempfile)) {
> + if (fsync_component(FSYNC_COMPONENT_PACKED_REFS, get_tempfile_fd(refs->tempfile)) ||
> + close_tempfile_gently(refs->tempfile)) {
> strbuf_addf(err, "error closing file %s: %s",
> get_tempfile_path(refs->tempfile),
> strerror(errno));
I do not necessarily agree with the organization to have it as a
component that is separate from other ref backends, but it is
very pleasing to see that there is only one fsync necessary for the
packed backend.
Nice.
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v5 3/5] core.fsync: introduce granular fsync control
2022-03-10 7:19 ` Junio C Hamano
@ 2022-03-10 18:38 ` Neeraj Singh
2022-03-10 18:44 ` Junio C Hamano
0 siblings, 1 reply; 122+ messages in thread
From: Neeraj Singh @ 2022-03-10 18:38 UTC (permalink / raw)
To: Junio C Hamano
Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker,
Bagas Sanjaya, Elijah Newren,
Ævar Arnfjörð Bjarmason, Patrick Steinhardt,
brian m. carlson, Neeraj K. Singh
On Wed, Mar 9, 2022 at 11:19 PM Junio C Hamano <gitster@pobox.com> wrote:
>
> Neeraj Singh <nksingh85@gmail.com> writes:
>
> > This is modeled after whitespace_rule_names. What if I change this to
> > the following?
> > static const struct fsync_component_name {
> > ...
> > } fsync_component_names[]
>
> That's better.
>
> > At the conclusion of this series, I defined 'default' as an aggregate
> > option that includes
> > the platform default. I'd prefer not to have any statefulness of the
> > core.fsync setting so
> > that there is less confusion about the final fsync configuration.
>
> Then scratch your preference ;-)
Just to clarify, linguistically, by 'scratch' do you mean that I should drop
my preference (which I can do to get the important parts of the
series in), or are you saying that that your suggestion is in line with
my preference, and I'm just not seeing it properly?
> Our configuration files are designed to be "hierarchical" in that
> system-wide default can be set in /etc/gitconfig, which can be
> overridden by per-user default in $HOME/.gitconfig, which can in
> turn be overridden by per-repository setting in .git/config, so
> starting from the compiled-in default, reading/augmenting "the value
> we tentatively decided based on what we have read so far" with what
> we read from lower-precedence files to higher-precedence files is a
> norm.
>
> Don't make this little corner of the system different from
> everything else; that will only confuse users.
>
> The git_config() callback should expect to see the same var with
> different values for that reason. Always restarting from zero will
> defeat it.
>
Consider core.whitespace. The parse_whitespace_rule code starts with
the compiled-in default every time it encounters a config value, and then
modifies it according to what the user passed in. So the user could figure
out what's going to happen by just looking at the value returned by
`git config --get core.whitespace`. The user doesn't need to call
`git config --get-all core.whitespace` and then reason about the entries
from top to bottom to figure out the actual state that Git will use.
> And always restarting from zero will mean "none" is meaningless,
> while it would be a quite nice way to say "forget everything we have
> read so far and start from scratch" when you really want to refuse
> what the system default wants to give you.
>
The intention, which I've implemented in my local v6 changes, is for an
empty list or empty string to be an illegal value of core.fsync. It should be
set to one or more legal values. But I see the advantage in always resetting to
the system default, like core.whitespace does, so that a set of unrecognized
values results in at least default behavior. An empty string would mean
'unconfigured', which will be meaningful when we integrate core.fsync with
core.fsyncObjectFiles.
I'll update the change to start from default, with none as a reset. I'm still in
favor of making it so that the most recent value of core.fsync in the
hierarchical configuration store stands alone without picking up state from
prior values.
> >> > @@ -1613,7 +1684,7 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
> >> > }
> >> >
> >> > if (!strcmp(var, "core.fsyncobjectfiles")) {
> >> > - fsync_object_files = git_config_bool(var, value);
> >> > + warning(_("core.fsyncobjectfiles is deprecated; use core.fsync instead"));
> >>
> >> This is not deprecating but removing the support, which I am not
> >> sure is a sensible thing to do. Rather we should pretend that
> >> core.fsync = "loose-object" (or "-loose-object") were found in the
> >> configuration, shouldn't we?
> >>
> >
> > The problem I anticipate is that figuring out the final configuration
> > becomes pretty
> > complex in the face of conflicting configurations of fsyncObjectFiles
> > and core.fsync.
>
> Don't start your thinking from too complex configuration that mixes
> and matches. Instead, think what happens to folks who are *NOT*
> interested in the new way of doing this. They aren't interested in
> setting core.fsync, and they already have core.fsyncObjectFiles set.
> You want to make sure their experience does not suck.
>
> The quoted code simply _IGNORES_ their wish and forces whatever
> default configuration core.fsync codepath happens to choose, which
> is a grave regression from their point of view.
>
> One way to handle this more gracefully is to delay the final
> decision until the end of the configuraiton file processing, and
> keep track of core.fsyncObjectFiles and core.fsync separately. If
> the latter is never set but the former is, then you are dealing with
> such a user who hasn't migrated. Give them a warning (the text
> above is fine---we can tell them "that's deprecated and you should
> use the other one instead"), but in the meantime, until deprecation
> turns into removal of support, keep honoring their original wish.
> If core.fsync is set to something, you can still give them a warning
> when you see core.fsyncObjectFiles, saying "that's deprecated and
> because you have core.fsync, we'll ignore the old one", and use the
> new method exclusively, without having to worry about mixing.
Is there a well-defined place where we know that configuration processing
is complete? The most obvious spot to me to integrate these two values would
be the first time we need to figure out the fsync state. Another spot would be
prepare_repo_settings. Are there any other good candidates?
Once the right spot is picked, I'll implement the integration of the
settings as you
suggested. For now I'll stick it in prepare_repo_settings.
Thanks for the review. Please expect a v6 today.
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH 6/8] core.fsync: add `fsync_component()` wrapper which doesn't die
2022-03-10 17:34 ` Junio C Hamano
@ 2022-03-10 18:40 ` Neeraj Singh
0 siblings, 0 replies; 122+ messages in thread
From: Neeraj Singh @ 2022-03-10 18:40 UTC (permalink / raw)
To: Junio C Hamano
Cc: Patrick Steinhardt, Git List, Randall S. Becker, Bagas Sanjaya,
Elijah Newren, Ævar Arnfjörð Bjarmason,
brian m. carlson, Neeraj K. Singh
On Thu, Mar 10, 2022 at 9:34 AM Junio C Hamano <gitster@pobox.com> wrote:
>
> Patrick Steinhardt <ps@pks.im> writes:
>
> > We have a `fsync_component_or_die()` helper function which only syncs
> > changes to disk in case the corresponding config is enabled by the user.
> > This wrapper will always die on an error though, which makes it
> > insufficient for new callsites we are about to add.
>
> You can replace "which makes it ..." part with a bit more concrete
> description to save suspense from the readers.
>
> fsync_component_or_die() that dies upon an error is not useful
> for callers with their own error handling or recovery logic,
> like ref transaction API.
>
> Split fsync_component() out that returns an error to help them.
>
> > -void fsync_or_die(int fd, const char *);
> > +int maybe_fsync(int fd);
> > ...
> > +static inline int fsync_component(enum fsync_component component, int fd)
> > +{
> > + if (fsync_components & component)
> > + return maybe_fsync(fd);
> > + return 0;
> > +}
> >
> > static inline void fsync_component_or_die(enum fsync_component component, int fd, const char *msg)
> > {
> > - if (fsync_components & component)
> > - fsync_or_die(fd, msg);
> > + if (fsync_component(component, fd) < 0)
> > + die_errno("fsync error on '%s'", msg);
> > }
>
> I think in the eventuall reroll, these "static inline" functions on
> the I/O code path will become real functions in write-or-die.c but
> other than that this reorganization looks sensible.
>
> Thanks.
>
Yes, that will be part of v6 of the base changeset.
> > diff --git a/write-or-die.c b/write-or-die.c
> > index 9faa5f9f56..4a5455ce46 100644
> > --- a/write-or-die.c
> > +++ b/write-or-die.c
> > @@ -56,19 +56,21 @@ void fprintf_or_die(FILE *f, const char *fmt, ...)
> > }
> > }
> >
> > -void fsync_or_die(int fd, const char *msg)
> > +int maybe_fsync(int fd)
> > {
> > if (use_fsync < 0)
> > use_fsync = git_env_bool("GIT_TEST_FSYNC", 1);
> > if (!use_fsync)
> > - return;
> > + return 0;
> >
> > if (fsync_method == FSYNC_METHOD_WRITEOUT_ONLY &&
> > git_fsync(fd, FSYNC_WRITEOUT_ONLY) >= 0)
> > - return;
> > + return 0;
> >
> > if (git_fsync(fd, FSYNC_HARDWARE_FLUSH) < 0)
> > - die_errno("fsync error on '%s'", msg);
> > + return -1;
> > +
> > + return 0;
> > }
> >
> > void write_or_die(int fd, const void *buf, size_t count)
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v5 3/5] core.fsync: introduce granular fsync control
2022-03-10 18:38 ` Neeraj Singh
@ 2022-03-10 18:44 ` Junio C Hamano
2022-03-10 19:57 ` Junio C Hamano
0 siblings, 1 reply; 122+ messages in thread
From: Junio C Hamano @ 2022-03-10 18:44 UTC (permalink / raw)
To: Neeraj Singh
Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker,
Bagas Sanjaya, Elijah Newren,
Ævar Arnfjörð Bjarmason, Patrick Steinhardt,
brian m. carlson, Neeraj K. Singh
Neeraj Singh <nksingh85@gmail.com> writes:
>> > At the conclusion of this series, I defined 'default' as an aggregate
>> > option that includes
>> > the platform default. I'd prefer not to have any statefulness of the
>> > core.fsync setting so
>> > that there is less confusion about the final fsync configuration.
>>
>> Then scratch your preference ;-)
>
> Just to clarify, linguistically, by 'scratch' do you mean that I should drop
> my preference
Yes.
> Is there a well-defined place where we know that configuration processing
> is complete? The most obvious spot to me to integrate these two values would
> be the first time we need to figure out the fsync state.
That sounds like a good place.
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH 7/8] core.fsync: new option to harden loose references
2022-03-10 18:25 ` Junio C Hamano
@ 2022-03-10 19:03 ` Neeraj Singh
0 siblings, 0 replies; 122+ messages in thread
From: Neeraj Singh @ 2022-03-10 19:03 UTC (permalink / raw)
To: Junio C Hamano
Cc: Patrick Steinhardt, Git List, Randall S. Becker, Bagas Sanjaya,
Elijah Newren, Ævar Arnfjörð Bjarmason,
brian m. carlson, Neeraj K. Singh
> Good illustration how the new helper in 6/8 is useful. It would be
> nice if the reroll of the base topic by Neeraj reorders the patches
> to introduce fsync_component() much earlier, at the same time it
> introduces the fsync_component_or_die().
Will do.
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v5 3/5] core.fsync: introduce granular fsync control
2022-03-10 18:44 ` Junio C Hamano
@ 2022-03-10 19:57 ` Junio C Hamano
2022-03-10 20:25 ` Neeraj Singh
0 siblings, 1 reply; 122+ messages in thread
From: Junio C Hamano @ 2022-03-10 19:57 UTC (permalink / raw)
To: Neeraj Singh
Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker,
Bagas Sanjaya, Elijah Newren,
Ævar Arnfjörð Bjarmason, Patrick Steinhardt,
brian m. carlson, Neeraj K. Singh
Junio C Hamano <gitster@pobox.com> writes:
> Neeraj Singh <nksingh85@gmail.com> writes:
>
>>> > At the conclusion of this series, I defined 'default' as an aggregate
>>> > option that includes
>>> > the platform default. I'd prefer not to have any statefulness of the
>>> > core.fsync setting so
>>> > that there is less confusion about the final fsync configuration.
>>>
>>> Then scratch your preference ;-)
>>
>> Just to clarify, linguistically, by 'scratch' do you mean that I should drop
>> my preference
>
> Yes.
Let me take this part back.
I do not mind too deeply if this were "each occurrence of core.fsync
as a whole replaces whatever we saw earlier, i.e. last-one-wins".
But if we were going that route, instead of starting from an empty
set, I'd prefer to see it begin with the built-in default (i.e. the
one you defined to mimic the traditional behaviour before core.fsync
was introduced) and added or deleted by each (possibly '-' prefixed)
element on the comma-separated list, with an explicit way to clear
the built-in default. E.g. "none,refs" would clear the components
traditionally fsync'ed by default and choose only "refs" component,
while "-pack-metadata" would mean the default ones minus
"pack-metadata" component are subject for fsync'ing. An empty
string would naturally mean "By having this core.fsync entry, I am
telling you not to pay any attention to what lower-precedence
configuration files said. But I want the built-in default, without
any additions or subtractions made by this entry, just the default,
please" in such a scheme, so do not forbid it.
Or, we can inherit from the previous configuration file to allow
/etc/gitconfig and the ones shipped by Git for Windows to augment
the built-in default before letting end-user configuration to
further customize the preference.
Either is fine by me.
Thanks.
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v5 3/5] core.fsync: introduce granular fsync control
2022-03-10 19:57 ` Junio C Hamano
@ 2022-03-10 20:25 ` Neeraj Singh
2022-03-10 21:17 ` Junio C Hamano
0 siblings, 1 reply; 122+ messages in thread
From: Neeraj Singh @ 2022-03-10 20:25 UTC (permalink / raw)
To: Junio C Hamano
Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker,
Bagas Sanjaya, Elijah Newren,
Ævar Arnfjörð Bjarmason, Patrick Steinhardt,
brian m. carlson, Neeraj K. Singh
On Thu, Mar 10, 2022 at 11:57 AM Junio C Hamano <gitster@pobox.com> wrote:
>
> Junio C Hamano <gitster@pobox.com> writes:
>
> > Neeraj Singh <nksingh85@gmail.com> writes:
> >
> >>> > At the conclusion of this series, I defined 'default' as an aggregate
> >>> > option that includes
> >>> > the platform default. I'd prefer not to have any statefulness of the
> >>> > core.fsync setting so
> >>> > that there is less confusion about the final fsync configuration.
> >>>
> >>> Then scratch your preference ;-)
> >>
> >> Just to clarify, linguistically, by 'scratch' do you mean that I should drop
> >> my preference
> >
> > Yes.
>
> Let me take this part back.
>
> I do not mind too deeply if this were "each occurrence of core.fsync
> as a whole replaces whatever we saw earlier, i.e. last-one-wins".
>
> But if we were going that route, instead of starting from an empty
> set, I'd prefer to see it begin with the built-in default (i.e. the
> one you defined to mimic the traditional behaviour before core.fsync
> was introduced) and added or deleted by each (possibly '-' prefixed)
> element on the comma-separated list, with an explicit way to clear
> the built-in default. E.g. "none,refs" would clear the components
> traditionally fsync'ed by default and choose only "refs" component,
> while "-pack-metadata" would mean the default ones minus
> "pack-metadata" component are subject for fsync'ing. An empty
> string would naturally mean "By having this core.fsync entry, I am
> telling you not to pay any attention to what lower-precedence
> configuration files said. But I want the built-in default, without
> any additions or subtractions made by this entry, just the default,
> please" in such a scheme, so do not forbid it.
>
> Or, we can inherit from the previous configuration file to allow
> /etc/gitconfig and the ones shipped by Git for Windows to augment
> the built-in default before letting end-user configuration to
> further customize the preference.
>
> Either is fine by me.
>
> Thanks.
Okay, I'll implement this version, since this is close to my preference.
Under this schema, 'default' isn't useful as an aggregate option, so
I'll eliminate
that.
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v5 3/5] core.fsync: introduce granular fsync control
2022-03-10 20:25 ` Neeraj Singh
@ 2022-03-10 21:17 ` Junio C Hamano
0 siblings, 0 replies; 122+ messages in thread
From: Junio C Hamano @ 2022-03-10 21:17 UTC (permalink / raw)
To: Neeraj Singh
Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker,
Bagas Sanjaya, Elijah Newren,
Ævar Arnfjörð Bjarmason, Patrick Steinhardt,
brian m. carlson, Neeraj K. Singh
Neeraj Singh <nksingh85@gmail.com> writes:
> Under this schema, 'default' isn't useful as an aggregate option, so
> I'll eliminate
> that.
Yeah, the only difference is the starting point. Either start with
default set of bits and give an option to clear, or start with an
empty set and give an option to set default. The former may be a
bit less cumbersome to users but there isn't a huge difference.
^ permalink raw reply [flat|nested] 122+ messages in thread
* [PATCH v6 0/6] A design for future-proofing fsync() configuration
2022-03-09 23:03 ` [PATCH v5 0/5] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget
` (8 preceding siblings ...)
2022-03-10 9:53 ` [PATCH 8/8] core.fsync: new option to harden packed references Patrick Steinhardt
@ 2022-03-10 22:43 ` Neeraj K. Singh via GitGitGadget
2022-03-10 22:43 ` [PATCH v6 1/6] wrapper: make inclusion of Windows csprng header tightly scoped Neeraj Singh via GitGitGadget
` (7 more replies)
9 siblings, 8 replies; 122+ messages in thread
From: Neeraj K. Singh via GitGitGadget @ 2022-03-10 22:43 UTC (permalink / raw)
To: git
Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps, sandals,
Johannes Schindelin, Neeraj K. Singh
This is an implementation of an extensible configuration mechanism for
fsyncing persistent components of a repo.
The main goals are to separate the "what" to sync from the "how". There are
now two settings: core.fsync - Control the 'what', including the index.
core.fsyncMethod - Control the 'how'. Currently we support writeout-only and
full fsync.
Syncing of refs can be layered on top of core.fsync. And batch mode will be
layered on core.fsyncMethod. Once this series reaches 'seen', I'll submit
ns/batched-fsync to introduce batch mode. Please see
https://github.com/gitgitgadget/git/pull/1134.
core.fsyncObjectfiles is removed and we will issue a deprecation warning if
it's seen.
I'd like to get agreement on this direction before submitting batch mode to
the list. The batch mode series is available to view at
Please see [1], [2], and [3] for discussions that led to this series.
After this change, new persistent data files added to the repo will need to
be added to the fsync_component enum and documented in the
Documentation/config/core.txt text.
V6 changes:
* Move only the windows csprng includes into wrapper.c rather than all of
them. This fixes the specific build issue due to broken Windows headers.
[6]
* Split the configuration parsing of core.fsync from the mechanism to focus
the review.
* Incorporate Patrick's patch at [7] into the core.fsync mechanism patch.
* Pick the stricter one of core.fsyncObjectFiles and (fsync_components &
FSYNC_COMPONENT_LOOSE_OBJECTS), to respect the older setting.
* Issue a deprecation warning but keep parsing and honoring
core.fsyncObjectFiles.
* Change configuration parsing of core.fsync to always start with the
platform default. none resets to the empty set. The comma separated list
implies a set without regards to ordering now. This follows Junio's
suggestion in [8].
* Change the documentation of the core.fsync option to reflect the way the
new parsing code works.
* The patch 7 and 8 of Patrick's series at [7] can be cherry-picked after
being applied to ns/core-fsyncmethod.
V5 changes:
* Rebase onto main at c2162907e9
* Add a patch to move CSPRNG platform includes to wrapper.c. This avoids
build errors in compat/win32/flush.c and other files.
* Move the documentation and aggregate options to the final patch in the
series.
* Define new aggregate options and guidance in line with Junio's suggestion
to present the user with 'levels of safety' rather than a morass of
detailed options.
V4 changes:
* Rebase onto master at b23dac905bd.
* Add a comment to write_pack_file indicating why we don't fsync when
writing to stdout.
* I kept the configuration schema as-is rather than switching to
multi-value. The thinking here is that a stateless last-one-wins config
schema (comma separated) will make it easier to achieve some holistic
self-consistent fsync configuration for a particular repo.
V3 changes:
* Remove relative path from git-compat-util.h include [4].
* Updated newly added warning texts to have more context for localization
[4].
* Fixed tab spacing in enum fsync_action
* Moved the fsync looping out to a helper and do it consistently. [4]
* Changed commit description to use camelCase for config names. [5]
* Add an optional fourth patch with derived-metadata so that the user can
exclude a forward-compatible set of things that should be recomputable
given existing data.
V2 changes:
* Updated the documentation for core.fsyncmethod to be less certain.
writeout-only probably does not do the right thing on Linux.
* Split out the core.fsync=index change into its own commit.
* Rename REPO_COMPONENT to FSYNC_COMPONENT. This is really specific to
fsyncing, so the name should reflect that.
* Re-add missing Makefile change for SYNC_FILE_RANGE.
* Tested writeout-only mode, index syncing, and general config settings.
[1] https://lore.kernel.org/git/211110.86r1bogg27.gmgdl@evledraar.gmail.com/
[2]
https://lore.kernel.org/git/dd65718814011eb93ccc4428f9882e0f025224a6.1636029491.git.ps@pks.im/
[3]
https://lore.kernel.org/git/pull.1076.git.git.1629856292.gitgitgadget@gmail.com/
[4]
https://lore.kernel.org/git/CANQDOdf8C4-haK9=Q_J4Cid8bQALnmGDm=SvatRbaVf+tkzqLw@mail.gmail.com/
[5] https://lore.kernel.org/git/211207.861r2opplg.gmgdl@evledraar.gmail.com/
[6]
https://lore.kernel.org/git/CANQDOdfZbOHZQt9Ah0t1AamTO2T7Gq0tmWX1jLqL6njE0LF6DA@mail.gmail.com/
[7]
https://lore.kernel.org/git/50e39f698a7c0cc06d3bc060e6dbc539ea693241.1646905589.git.ps@pks.im/
[8] https://lore.kernel.org/git/xmqqk0d1cxsv.fsf@gitster.g/
Neeraj Singh (6):
wrapper: make inclusion of Windows csprng header tightly scoped
core.fsyncmethod: add writeout-only mode
core.fsync: introduce granular fsync control infrastructure
core.fsync: add configuration parsing
core.fsync: new option to harden the index
core.fsync: documentation and user-friendly aggregate options
Documentation/config/core.txt | 58 ++++++++++++++++--
Makefile | 6 ++
builtin/fast-import.c | 2 +-
builtin/index-pack.c | 4 +-
builtin/pack-objects.c | 24 +++++---
bulk-checkin.c | 5 +-
cache.h | 48 +++++++++++++++
commit-graph.c | 3 +-
compat/mingw.h | 3 +
compat/win32/flush.c | 28 +++++++++
compat/winansi.c | 5 --
config.c | 94 +++++++++++++++++++++++++++++
config.mak.uname | 3 +
configure.ac | 8 +++
contrib/buildsystems/CMakeLists.txt | 16 +++--
csum-file.c | 5 +-
csum-file.h | 3 +-
environment.c | 4 +-
git-compat-util.h | 30 +++++++--
midx.c | 3 +-
object-file.c | 13 ++--
pack-bitmap-write.c | 3 +-
pack-write.c | 13 ++--
read-cache.c | 19 ++++--
wrapper.c | 71 ++++++++++++++++++++++
write-or-die.c | 33 ++++++++--
26 files changed, 444 insertions(+), 60 deletions(-)
create mode 100644 compat/win32/flush.c
base-commit: c2162907e9aa884bdb70208389cb99b181620d51
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1093%2Fneerajsi-msft%2Fns%2Fcore-fsync-v6
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1093/neerajsi-msft/ns/core-fsync-v6
Pull-Request: https://github.com/gitgitgadget/git/pull/1093
Range-diff vs v5:
1: 685b1db8880 ! 1: 825079b6aa1 wrapper: move inclusion of CSPRNG headers the wrapper.c file
@@ Metadata
Author: Neeraj Singh <neerajsi@microsoft.com>
## Commit message ##
- wrapper: move inclusion of CSPRNG headers the wrapper.c file
+ wrapper: make inclusion of Windows csprng header tightly scoped
Including NTSecAPI.h in git-compat-util.h causes build errors in any
- other file that includes winternl.h. That file was included in order to
+ other file that includes winternl.h. NTSecAPI.h was included in order to
get access to the RtlGenRandom cryptographically secure PRNG. This
- change scopes the inclusion of all PRNG headers to just the wrapper.c
- file, which is the only place it is really needed.
+ change scopes the inclusion of ntsecapi.h to wrapper.c, which is the only
+ place that it's actually needed.
+
+ The build breakage is due to the definition of UNICODE_STRING in
+ NtSecApi.h:
+ #ifndef _NTDEF_
+ typedef LSA_UNICODE_STRING UNICODE_STRING, *PUNICODE_STRING;
+ typedef LSA_STRING STRING, *PSTRING ;
+ #endif
+
+ LsaLookup.h:
+ typedef struct _LSA_UNICODE_STRING {
+ USHORT Length;
+ USHORT MaximumLength;
+ #ifdef MIDL_PASS
+ [size_is(MaximumLength/2), length_is(Length/2)]
+ #endif // MIDL_PASS
+ PWSTR Buffer;
+ } LSA_UNICODE_STRING, *PLSA_UNICODE_STRING;
+
+ winternl.h also defines UNICODE_STRING:
+ typedef struct _UNICODE_STRING {
+ USHORT Length;
+ USHORT MaximumLength;
+ PWSTR Buffer;
+ } UNICODE_STRING;
+ typedef UNICODE_STRING *PUNICODE_STRING;
+
+ Both definitions have equivalent layouts. Apparently these internal
+ Windows headers aren't designed to be included together. This is
+ an oversight in the headers and does not represent an incompatibility
+ between the APIs.
Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
@@ git-compat-util.h
#endif
#include <unistd.h>
-@@
- #else
- #include <stdint.h>
- #endif
--#ifdef HAVE_ARC4RANDOM_LIBBSD
--#include <bsd/stdlib.h>
--#endif
--#ifdef HAVE_GETRANDOM
--#include <sys/random.h>
--#endif
- #ifdef NO_INTPTR_T
- /*
- * On I16LP32, ILP32 and LP64 "long" is the safe bet, however
## wrapper.c ##
@@
@@ wrapper.c
+#include <NTSecAPI.h>
+#undef SystemFunction036
+#endif
-+
-+#ifdef HAVE_ARC4RANDOM_LIBBSD
-+#include <bsd/stdlib.h>
-+#endif
-+#ifdef HAVE_GETRANDOM
-+#include <sys/random.h>
-+#endif
+
static int memory_limit_check(size_t size, int gentle)
{
2: da8cfc10bb4 = 2: a41bd4a06af core.fsyncmethod: add writeout-only mode
3: e31886717b4 ! 3: 64e2bdcdfd9 core.fsync: introduce granular fsync control
@@ Metadata
Author: Neeraj Singh <neerajsi@microsoft.com>
## Commit message ##
- core.fsync: introduce granular fsync control
+ core.fsync: introduce granular fsync control infrastructure
- This commit introduces the `core.fsync` configuration
- knob which can be used to control how components of the
- repository are made durable on disk.
+ This commit introduces the infrastructure for the core.fsync
+ configuration knob. The repository components we want to sync
+ are identified by flags so that we can turn on or off syncing
+ for specific components.
- This setting allows future extensibility of the list of
- syncable components:
- * We issue a warning rather than an error for unrecognized
- components, so new configs can be used with old Git versions.
- * We support negation, so users can choose one of the aggregate
- options and then remove components that they don't want.
- Aggregate options are defined in a later patch in this series.
+ If core.fsyncObjectFiles is set and the core.fsync configuration
+ also includes FSYNC_COMPONENT_LOOSE_OBJECT, we will fsync any
+ loose objects. This picks the strictest data integrity behavior
+ if core.fsync and core.fsyncObjectFiles are set to conflicting values.
- This also supports the common request of doing absolutely no
- fysncing with the `core.fsync=none` value, which is expected
- to make the test suite faster.
+ This change introduces the currently unused fsync_component
+ helper, which will be used by a later patch that adds fsyncing to
+ the refs backend.
- Complete documentation for the new setting is included in a later patch
- in the series so that it can be reviewed in final form.
+ Actual configuration and documentation of the fsync components
+ list are in other patches in the series to separate review of
+ the underlying mechanism from the policy of how it's configured.
+ Helped-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
- ## Documentation/config/core.txt ##
-@@ Documentation/config/core.txt: core.fsyncMethod::
- filesystem and storage hardware, data added to the repository may not be
- durable in the event of a system crash. This is the default mode on macOS.
-
--core.fsyncObjectFiles::
-- This boolean will enable 'fsync()' when writing object files.
--+
--This is a total waste of time and effort on a filesystem that orders
--data writes properly, but can be useful for filesystems that do not use
--journalling (traditional UNIX filesystems) or that only journal metadata
--and not file contents (OS X's HFS+, or Linux ext3 with "data=writeback").
--
- core.preloadIndex::
- Enable parallel index preload for operations like 'git diff'
- +
-
## builtin/fast-import.c ##
@@ builtin/fast-import.c: static void end_packfile(void)
struct tag *t;
@@ cache.h: void reset_shared_repository(void);
extern int read_replace_refs;
extern char *git_replace_ref_base;
--extern int fsync_object_files;
--extern int use_fsync;
+/*
+ * These values are used to help identify parts of a repository to fsync.
+ * FSYNC_COMPONENT_NONE identifies data that will not be a persistent part of the
@@ cache.h: void reset_shared_repository(void);
+ * A bitmask indicating which components of the repo should be fsynced.
+ */
+extern enum fsync_component fsync_components;
+ extern int fsync_object_files;
+ extern int use_fsync;
- enum fsync_method {
- FSYNC_METHOD_FSYNC,
-@@ cache.h: enum fsync_method {
- };
-
- extern enum fsync_method fsync_method;
-+extern int use_fsync;
- extern int core_preload_index;
- extern int precomposed_unicode;
- extern int protect_hfs;
@@ cache.h: int copy_file_with_time(const char *dst, const char *src, int mode);
+
void write_or_die(int fd, const void *buf, size_t count);
void fsync_or_die(int fd, const char *);
++int fsync_component(enum fsync_component component, int fd);
++void fsync_component_or_die(enum fsync_component component, int fd, const char *msg);
-+static inline void fsync_component_or_die(enum fsync_component component, int fd, const char *msg)
-+{
-+ if (fsync_components & component)
-+ fsync_or_die(fd, msg);
-+}
-+
ssize_t read_in_full(int fd, void *buf, size_t count);
ssize_t write_in_full(int fd, const void *buf, size_t count);
- ssize_t pread_in_full(int fd, void *buf, size_t count, off_t offset);
## commit-graph.c ##
@@ commit-graph.c: static int write_commit_graph_file(struct write_commit_graph_context *ctx)
@@ commit-graph.c: static int write_commit_graph_file(struct write_commit_graph_con
if (ctx->split) {
- ## config.c ##
-@@ config.c: static int git_parse_maybe_bool_text(const char *value)
- return -1;
- }
-
-+static const struct fsync_component_entry {
-+ const char *name;
-+ enum fsync_component component_bits;
-+} fsync_component_table[] = {
-+ { "loose-object", FSYNC_COMPONENT_LOOSE_OBJECT },
-+ { "pack", FSYNC_COMPONENT_PACK },
-+ { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA },
-+ { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH },
-+};
-+
-+static enum fsync_component parse_fsync_components(const char *var, const char *string)
-+{
-+ enum fsync_component output = 0;
-+
-+ if (!strcmp(string, "none"))
-+ return FSYNC_COMPONENT_NONE;
-+
-+ while (string) {
-+ int i;
-+ size_t len;
-+ const char *ep;
-+ int negated = 0;
-+ int found = 0;
-+
-+ string = string + strspn(string, ", \t\n\r");
-+ ep = strchrnul(string, ',');
-+ len = ep - string;
-+
-+ if (*string == '-') {
-+ negated = 1;
-+ string++;
-+ len--;
-+ if (!len)
-+ warning(_("invalid value for variable %s"), var);
-+ }
-+
-+ if (!len)
-+ break;
-+
-+ for (i = 0; i < ARRAY_SIZE(fsync_component_table); ++i) {
-+ const struct fsync_component_entry *entry = &fsync_component_table[i];
-+
-+ if (strncmp(entry->name, string, len))
-+ continue;
-+
-+ found = 1;
-+ if (negated)
-+ output &= ~entry->component_bits;
-+ else
-+ output |= entry->component_bits;
-+ }
-+
-+ if (!found) {
-+ char *component = xstrndup(string, len);
-+ warning(_("ignoring unknown core.fsync component '%s'"), component);
-+ free(component);
-+ }
-+
-+ string = ep;
-+ }
-+
-+ return output;
-+}
-+
- int git_parse_maybe_bool(const char *value)
- {
- int v = git_parse_maybe_bool_text(value);
-@@ config.c: static int git_default_core_config(const char *var, const char *value, void *cb)
- return 0;
- }
-
-+ if (!strcmp(var, "core.fsync")) {
-+ if (!value)
-+ return config_error_nonbool(var);
-+ fsync_components = parse_fsync_components(var, value);
-+ return 0;
-+ }
-+
- if (!strcmp(var, "core.fsyncmethod")) {
- if (!value)
- return config_error_nonbool(var);
-@@ config.c: static int git_default_core_config(const char *var, const char *value, void *cb)
- }
-
- if (!strcmp(var, "core.fsyncobjectfiles")) {
-- fsync_object_files = git_config_bool(var, value);
-+ warning(_("core.fsyncobjectfiles is deprecated; use core.fsync instead"));
- return 0;
- }
-
-
## csum-file.c ##
@@ csum-file.c: static void free_hashfile(struct hashfile *f)
free(f);
@@ csum-file.h: int hashfile_truncate(struct hashfile *, struct hashfile_checkpoint
void crc32_begin(struct hashfile *);
## environment.c ##
-@@ environment.c: const char *git_attributes_file;
- const char *git_hooks_path;
- int zlib_compression_level = Z_BEST_SPEED;
- int pack_compression_level = Z_DEFAULT_COMPRESSION;
--int fsync_object_files;
+@@ environment.c: int pack_compression_level = Z_DEFAULT_COMPRESSION;
+ int fsync_object_files;
int use_fsync = -1;
enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT;
+enum fsync_component fsync_components = FSYNC_COMPONENTS_DEFAULT;
@@ midx.c: static int write_midx_internal(const char *object_dir,
## object-file.c ##
@@ object-file.c: int hash_object_file(const struct git_hash_algo *algo, const void *buf,
+ /* Finalize a file on disk, and close it. */
static void close_loose_object(int fd)
{
- if (!the_repository->objects->odb->will_destroy) {
+- if (!the_repository->objects->odb->will_destroy) {
- if (fsync_object_files)
- fsync_or_die(fd, "loose object file");
-+ fsync_component_or_die(FSYNC_COMPONENT_LOOSE_OBJECT, fd, "loose object file");
- }
+- }
++ if (the_repository->objects->odb->will_destroy)
++ goto out;
++ if (fsync_object_files > 0)
++ fsync_or_die(fd, "loose object file");
++ else
++ fsync_component_or_die(FSYNC_COMPONENT_LOOSE_OBJECT, fd,
++ "loose object file");
++
++out:
if (close(fd) != 0)
+ die_errno(_("error when closing loose object file"));
+ }
## pack-bitmap-write.c ##
@@ pack-bitmap-write.c: void bitmap_writer_finish(struct pack_idx_entry **index,
@@ read-cache.c: static int do_write_index(struct index_state *istate, struct tempf
if (close_tempfile_gently(tempfile)) {
error(_("could not close '%s'"), get_tempfile_path(tempfile));
return -1;
+
+ ## write-or-die.c ##
+@@ write-or-die.c: void fprintf_or_die(FILE *f, const char *fmt, ...)
+ }
+ }
+
+-void fsync_or_die(int fd, const char *msg)
++static int maybe_fsync(int fd)
+ {
+ if (use_fsync < 0)
+ use_fsync = git_env_bool("GIT_TEST_FSYNC", 1);
+ if (!use_fsync)
+- return;
++ return 0;
+
+ if (fsync_method == FSYNC_METHOD_WRITEOUT_ONLY &&
+ git_fsync(fd, FSYNC_WRITEOUT_ONLY) >= 0)
+- return;
++ return 0;
++
++ return git_fsync(fd, FSYNC_HARDWARE_FLUSH);
++}
+
+- if (git_fsync(fd, FSYNC_HARDWARE_FLUSH) < 0)
++void fsync_or_die(int fd, const char *msg)
++{
++ if (maybe_fsync(fd) < 0)
+ die_errno("fsync error on '%s'", msg);
+ }
+
++int fsync_component(enum fsync_component component, int fd)
++{
++ if (fsync_components & component)
++ return maybe_fsync(fd);
++ return 0;
++}
++
++void fsync_component_or_die(enum fsync_component component, int fd, const char *msg)
++{
++ if (fsync_components & component)
++ fsync_or_die(fd, msg);
++}
++
+ void write_or_die(int fd, const void *buf, size_t count)
+ {
+ if (write_in_full(fd, buf, count) < 0) {
-: ----------- > 4: 6adc8dc1385 core.fsync: add configuration parsing
4: 9da808ba743 ! 5: 757f6d0bbd2 core.fsync: new option to harden the index
@@ cache.h: enum fsync_component {
#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \
## config.c ##
-@@ config.c: static const struct fsync_component_entry {
+@@ config.c: static const struct fsync_component_name {
{ "pack", FSYNC_COMPONENT_PACK },
{ "pack-metadata", FSYNC_COMPONENT_PACK_METADATA },
{ "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH },
5: 2d71346b10e ! 6: 7e4cc6e10a5 core.fsync: documentation and user-friendly aggregate options
@@ Documentation/config/core.txt: core.whitespace::
errors. The default tab width is 8. Allowed values are 1 to 63.
+core.fsync::
-+ A comma-separated list of parts of the repository which should be
-+ hardened via the core.fsyncMethod when created or modified. You can
-+ disable hardening of any component by prefixing it with a '-'. Later
-+ items take precedence over earlier ones in the comma-separated list.
-+ For example, `core.fsync=all,-pack-metadata` means "harden everything
-+ except pack metadata." Items that are not hardened may be lost in the
-+ event of an unclean system shutdown. Unless you have special
-+ requirements, it is recommended that you leave this option as default
-+ or pick one of `committed`, `added`, or `all`.
++ A comma-separated list of components of the repository that
++ should be hardened via the core.fsyncMethod when created or
++ modified. You can disable hardening of any component by
++ prefixing it with a '-'. Items that are not hardened may be
++ lost in the event of an unclean system shutdown. Unless you
++ have special requirements, it is recommended that you leave
++ this option empty or pick one of `committed`, `added`,
++ or `all`.
++
-+* `none` disables fsync completely. This value must be specified alone.
++When this configuration is encountered, the set of components starts with
++the platform default value, disabled components are removed, and additional
++components are added. `none` resets the state so that the platform default
++is ignored.
+++
++The empty string resets the fsync configuration to the platform
++default. The platform default on most platform is equivalent to
++`core.fsync=committed,-loose-object`, which has good performance,
++but risks losing recent work in the event of an unclean system shutdown.
+++
++* `none` clears the set of fsynced components.
+* `loose-object` hardens objects added to the repo in loose-object form.
+* `pack` hardens objects added to the repo in packfile form.
+* `pack-metadata` hardens packfile bitmaps and indexes.
@@ Documentation/config/core.txt: core.whitespace::
+ `loose-object,pack`.
+* `derived-metadata` is an aggregate option that is equivalent to
+ `pack-metadata,commit-graph`.
-+* `default` is an aggregate option that is equivalent to
-+ `objects,derived-metadata,-loose-object`. This mode is enabled by default.
-+ It has good performance, but risks losing recent work if the system shuts
-+ down uncleanly, since commits, trees, and blobs in loose-object form may be
-+ lost.
+* `committed` is an aggregate option that is currently equivalent to
-+ `objects`. This mode sacrifices some performance to ensure that all work
++ `objects`. This mode sacrifices some performance to ensure that work
+ that is committed to the repository with `git commit` or similar commands
-+ is preserved.
++ is hardened.
+* `added` is an aggregate option that is currently equivalent to
+ `committed,index`. This mode sacrifices additional performance to
+ ensure that the results of commands like `git add` and similar operations
-+ are preserved.
++ are hardened.
+* `all` is an aggregate option that syncs all individual components above.
+
core.fsyncMethod::
@@ cache.h: enum fsync_component {
* A bitmask indicating which components of the repo should be fsynced.
## config.c ##
-@@ config.c: static const struct fsync_component_entry {
+@@ config.c: static const struct fsync_component_name {
{ "pack-metadata", FSYNC_COMPONENT_PACK_METADATA },
{ "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH },
{ "index", FSYNC_COMPONENT_INDEX },
+ { "objects", FSYNC_COMPONENTS_OBJECTS },
+ { "derived-metadata", FSYNC_COMPONENTS_DERIVED_METADATA },
-+ { "default", FSYNC_COMPONENTS_DEFAULT },
+ { "committed", FSYNC_COMPONENTS_COMMITTED },
+ { "added", FSYNC_COMPONENTS_ADDED },
+ { "all", FSYNC_COMPONENTS_ALL },
--
gitgitgadget
^ permalink raw reply [flat|nested] 122+ messages in thread
* [PATCH v6 1/6] wrapper: make inclusion of Windows csprng header tightly scoped
2022-03-10 22:43 ` [PATCH v6 0/6] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget
@ 2022-03-10 22:43 ` Neeraj Singh via GitGitGadget
2022-03-10 22:43 ` [PATCH v6 2/6] core.fsyncmethod: add writeout-only mode Neeraj Singh via GitGitGadget
` (6 subsequent siblings)
7 siblings, 0 replies; 122+ messages in thread
From: Neeraj Singh via GitGitGadget @ 2022-03-10 22:43 UTC (permalink / raw)
To: git
Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps, sandals,
Johannes Schindelin, Neeraj K. Singh, Neeraj Singh
From: Neeraj Singh <neerajsi@microsoft.com>
Including NTSecAPI.h in git-compat-util.h causes build errors in any
other file that includes winternl.h. NTSecAPI.h was included in order to
get access to the RtlGenRandom cryptographically secure PRNG. This
change scopes the inclusion of ntsecapi.h to wrapper.c, which is the only
place that it's actually needed.
The build breakage is due to the definition of UNICODE_STRING in
NtSecApi.h:
#ifndef _NTDEF_
typedef LSA_UNICODE_STRING UNICODE_STRING, *PUNICODE_STRING;
typedef LSA_STRING STRING, *PSTRING ;
#endif
LsaLookup.h:
typedef struct _LSA_UNICODE_STRING {
USHORT Length;
USHORT MaximumLength;
#ifdef MIDL_PASS
[size_is(MaximumLength/2), length_is(Length/2)]
#endif // MIDL_PASS
PWSTR Buffer;
} LSA_UNICODE_STRING, *PLSA_UNICODE_STRING;
winternl.h also defines UNICODE_STRING:
typedef struct _UNICODE_STRING {
USHORT Length;
USHORT MaximumLength;
PWSTR Buffer;
} UNICODE_STRING;
typedef UNICODE_STRING *PUNICODE_STRING;
Both definitions have equivalent layouts. Apparently these internal
Windows headers aren't designed to be included together. This is
an oversight in the headers and does not represent an incompatibility
between the APIs.
Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
---
compat/winansi.c | 5 -----
git-compat-util.h | 6 ------
wrapper.c | 7 +++++++
3 files changed, 7 insertions(+), 11 deletions(-)
diff --git a/compat/winansi.c b/compat/winansi.c
index 936a80a5f00..3abe8dd5a27 100644
--- a/compat/winansi.c
+++ b/compat/winansi.c
@@ -4,11 +4,6 @@
#undef NOGDI
-/*
- * Including the appropriate header file for RtlGenRandom causes MSVC to see a
- * redefinition of types in an incompatible way when including headers below.
- */
-#undef HAVE_RTLGENRANDOM
#include "../git-compat-util.h"
#include <wingdi.h>
#include <winreg.h>
diff --git a/git-compat-util.h b/git-compat-util.h
index 876907b9df4..d210cff058c 100644
--- a/git-compat-util.h
+++ b/git-compat-util.h
@@ -197,12 +197,6 @@
#endif
#include <windows.h>
#define GIT_WINDOWS_NATIVE
-#ifdef HAVE_RTLGENRANDOM
-/* This is required to get access to RtlGenRandom. */
-#define SystemFunction036 NTAPI SystemFunction036
-#include <NTSecAPI.h>
-#undef SystemFunction036
-#endif
#endif
#include <unistd.h>
diff --git a/wrapper.c b/wrapper.c
index 3258cdb171f..1108e4840a4 100644
--- a/wrapper.c
+++ b/wrapper.c
@@ -4,6 +4,13 @@
#include "cache.h"
#include "config.h"
+#ifdef HAVE_RTLGENRANDOM
+/* This is required to get access to RtlGenRandom. */
+#define SystemFunction036 NTAPI SystemFunction036
+#include <NTSecAPI.h>
+#undef SystemFunction036
+#endif
+
static int memory_limit_check(size_t size, int gentle)
{
static size_t limit = 0;
--
gitgitgadget
^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v6 2/6] core.fsyncmethod: add writeout-only mode
2022-03-10 22:43 ` [PATCH v6 0/6] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget
2022-03-10 22:43 ` [PATCH v6 1/6] wrapper: make inclusion of Windows csprng header tightly scoped Neeraj Singh via GitGitGadget
@ 2022-03-10 22:43 ` Neeraj Singh via GitGitGadget
2022-03-10 22:43 ` [PATCH v6 3/6] core.fsync: introduce granular fsync control infrastructure Neeraj Singh via GitGitGadget
` (5 subsequent siblings)
7 siblings, 0 replies; 122+ messages in thread
From: Neeraj Singh via GitGitGadget @ 2022-03-10 22:43 UTC (permalink / raw)
To: git
Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps, sandals,
Johannes Schindelin, Neeraj K. Singh, Neeraj Singh
From: Neeraj Singh <neerajsi@microsoft.com>
This commit introduces the `core.fsyncMethod` configuration
knob, which can currently be set to `fsync` or `writeout-only`.
The new writeout-only mode attempts to tell the operating system to
flush its in-memory page cache to the storage hardware without issuing a
CACHE_FLUSH command to the storage controller.
Writeout-only fsync is significantly faster than a vanilla fsync on
common hardware, since data is written to a disk-side cache rather than
all the way to a durable medium. Later changes in this patch series will
take advantage of this primitive to implement batching of hardware
flushes.
When git_fsync is called with FSYNC_WRITEOUT_ONLY, it may fail and the
caller is expected to do an ordinary fsync as needed.
On Apple platforms, the fsync system call does not issue a CACHE_FLUSH
directive to the storage controller. This change updates fsync to do
fcntl(F_FULLFSYNC) to make fsync actually durable. We maintain parity
with existing behavior on Apple platforms by setting the default value
of the new core.fsyncMethod option.
Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
---
Documentation/config/core.txt | 9 ++++
Makefile | 6 +++
cache.h | 7 ++++
compat/mingw.h | 3 ++
compat/win32/flush.c | 28 +++++++++++++
config.c | 12 ++++++
config.mak.uname | 3 ++
configure.ac | 8 ++++
contrib/buildsystems/CMakeLists.txt | 16 ++++++--
environment.c | 1 +
git-compat-util.h | 24 +++++++++++
wrapper.c | 64 +++++++++++++++++++++++++++++
write-or-die.c | 11 +++--
13 files changed, 184 insertions(+), 8 deletions(-)
create mode 100644 compat/win32/flush.c
diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
index c04f62a54a1..dbb134f7136 100644
--- a/Documentation/config/core.txt
+++ b/Documentation/config/core.txt
@@ -547,6 +547,15 @@ core.whitespace::
is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent`
errors. The default tab width is 8. Allowed values are 1 to 63.
+core.fsyncMethod::
+ A value indicating the strategy Git will use to harden repository data
+ using fsync and related primitives.
++
+* `fsync` uses the fsync() system call or platform equivalents.
+* `writeout-only` issues pagecache writeback requests, but depending on the
+ filesystem and storage hardware, data added to the repository may not be
+ durable in the event of a system crash. This is the default mode on macOS.
+
core.fsyncObjectFiles::
This boolean will enable 'fsync()' when writing object files.
+
diff --git a/Makefile b/Makefile
index 6f0b4b775fe..17fd9b023a4 100644
--- a/Makefile
+++ b/Makefile
@@ -411,6 +411,8 @@ all::
#
# Define HAVE_CLOCK_MONOTONIC if your platform has CLOCK_MONOTONIC.
#
+# Define HAVE_SYNC_FILE_RANGE if your platform has sync_file_range.
+#
# Define NEEDS_LIBRT if your platform requires linking with librt (glibc version
# before 2.17) for clock_gettime and CLOCK_MONOTONIC.
#
@@ -1897,6 +1899,10 @@ ifdef HAVE_CLOCK_MONOTONIC
BASIC_CFLAGS += -DHAVE_CLOCK_MONOTONIC
endif
+ifdef HAVE_SYNC_FILE_RANGE
+ BASIC_CFLAGS += -DHAVE_SYNC_FILE_RANGE
+endif
+
ifdef NEEDS_LIBRT
EXTLIBS += -lrt
endif
diff --git a/cache.h b/cache.h
index 04d4d2db25c..82f0194a3dd 100644
--- a/cache.h
+++ b/cache.h
@@ -995,6 +995,13 @@ extern char *git_replace_ref_base;
extern int fsync_object_files;
extern int use_fsync;
+
+enum fsync_method {
+ FSYNC_METHOD_FSYNC,
+ FSYNC_METHOD_WRITEOUT_ONLY
+};
+
+extern enum fsync_method fsync_method;
extern int core_preload_index;
extern int precomposed_unicode;
extern int protect_hfs;
diff --git a/compat/mingw.h b/compat/mingw.h
index c9a52ad64a6..6074a3d3ced 100644
--- a/compat/mingw.h
+++ b/compat/mingw.h
@@ -329,6 +329,9 @@ int mingw_getpagesize(void);
#define getpagesize mingw_getpagesize
#endif
+int win32_fsync_no_flush(int fd);
+#define fsync_no_flush win32_fsync_no_flush
+
struct rlimit {
unsigned int rlim_cur;
};
diff --git a/compat/win32/flush.c b/compat/win32/flush.c
new file mode 100644
index 00000000000..291f90ea940
--- /dev/null
+++ b/compat/win32/flush.c
@@ -0,0 +1,28 @@
+#include "git-compat-util.h"
+#include <winternl.h>
+#include "lazyload.h"
+
+int win32_fsync_no_flush(int fd)
+{
+ IO_STATUS_BLOCK io_status;
+
+#define FLUSH_FLAGS_FILE_DATA_ONLY 1
+
+ DECLARE_PROC_ADDR(ntdll.dll, NTSTATUS, NTAPI, NtFlushBuffersFileEx,
+ HANDLE FileHandle, ULONG Flags, PVOID Parameters, ULONG ParameterSize,
+ PIO_STATUS_BLOCK IoStatusBlock);
+
+ if (!INIT_PROC_ADDR(NtFlushBuffersFileEx)) {
+ errno = ENOSYS;
+ return -1;
+ }
+
+ memset(&io_status, 0, sizeof(io_status));
+ if (NtFlushBuffersFileEx((HANDLE)_get_osfhandle(fd), FLUSH_FLAGS_FILE_DATA_ONLY,
+ NULL, 0, &io_status)) {
+ errno = EINVAL;
+ return -1;
+ }
+
+ return 0;
+}
diff --git a/config.c b/config.c
index 383b1a4885b..f3ff80b01c9 100644
--- a/config.c
+++ b/config.c
@@ -1600,6 +1600,18 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
return 0;
}
+ if (!strcmp(var, "core.fsyncmethod")) {
+ if (!value)
+ return config_error_nonbool(var);
+ if (!strcmp(value, "fsync"))
+ fsync_method = FSYNC_METHOD_FSYNC;
+ else if (!strcmp(value, "writeout-only"))
+ fsync_method = FSYNC_METHOD_WRITEOUT_ONLY;
+ else
+ warning(_("ignoring unknown core.fsyncMethod value '%s'"), value);
+
+ }
+
if (!strcmp(var, "core.fsyncobjectfiles")) {
fsync_object_files = git_config_bool(var, value);
return 0;
diff --git a/config.mak.uname b/config.mak.uname
index 4352ea39e9b..404fff5dd04 100644
--- a/config.mak.uname
+++ b/config.mak.uname
@@ -57,6 +57,7 @@ ifeq ($(uname_S),Linux)
HAVE_CLOCK_MONOTONIC = YesPlease
# -lrt is needed for clock_gettime on glibc <= 2.16
NEEDS_LIBRT = YesPlease
+ HAVE_SYNC_FILE_RANGE = YesPlease
HAVE_GETDELIM = YesPlease
FREAD_READS_DIRECTORIES = UnfortunatelyYes
BASIC_CFLAGS += -DHAVE_SYSINFO
@@ -463,6 +464,7 @@ endif
CFLAGS =
BASIC_CFLAGS = -nologo -I. -Icompat/vcbuild/include -DWIN32 -D_CONSOLE -DHAVE_STRING_H -D_CRT_SECURE_NO_WARNINGS -D_CRT_NONSTDC_NO_DEPRECATE
COMPAT_OBJS = compat/msvc.o compat/winansi.o \
+ compat/win32/flush.o \
compat/win32/path-utils.o \
compat/win32/pthread.o compat/win32/syslog.o \
compat/win32/trace2_win32_process_info.o \
@@ -640,6 +642,7 @@ ifeq ($(uname_S),MINGW)
COMPAT_CFLAGS += -DSTRIP_EXTENSION=\".exe\"
COMPAT_OBJS += compat/mingw.o compat/winansi.o \
compat/win32/trace2_win32_process_info.o \
+ compat/win32/flush.o \
compat/win32/path-utils.o \
compat/win32/pthread.o compat/win32/syslog.o \
compat/win32/dirent.o
diff --git a/configure.ac b/configure.ac
index 5ee25ec95c8..6bd6bef1c44 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1082,6 +1082,14 @@ AC_COMPILE_IFELSE([CLOCK_MONOTONIC_SRC],
[AC_MSG_RESULT([no])
HAVE_CLOCK_MONOTONIC=])
GIT_CONF_SUBST([HAVE_CLOCK_MONOTONIC])
+
+#
+# Define HAVE_SYNC_FILE_RANGE=YesPlease if sync_file_range is available.
+GIT_CHECK_FUNC(sync_file_range,
+ [HAVE_SYNC_FILE_RANGE=YesPlease],
+ [HAVE_SYNC_FILE_RANGE])
+GIT_CONF_SUBST([HAVE_SYNC_FILE_RANGE])
+
#
# Define NO_SETITIMER if you don't have setitimer.
GIT_CHECK_FUNC(setitimer,
diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt
index e44232f85d3..3a9e6241660 100644
--- a/contrib/buildsystems/CMakeLists.txt
+++ b/contrib/buildsystems/CMakeLists.txt
@@ -261,10 +261,18 @@ if(CMAKE_SYSTEM_NAME STREQUAL "Windows")
NOGDI OBJECT_CREATION_MODE=1 __USE_MINGW_ANSI_STDIO=0
USE_NED_ALLOCATOR OVERRIDE_STRDUP MMAP_PREVENTS_DELETE USE_WIN32_MMAP
UNICODE _UNICODE HAVE_WPGMPTR ENSURE_MSYSTEM_IS_SET HAVE_RTLGENRANDOM)
- list(APPEND compat_SOURCES compat/mingw.c compat/winansi.c compat/win32/path-utils.c
- compat/win32/pthread.c compat/win32mmap.c compat/win32/syslog.c
- compat/win32/trace2_win32_process_info.c compat/win32/dirent.c
- compat/nedmalloc/nedmalloc.c compat/strdup.c)
+ list(APPEND compat_SOURCES
+ compat/mingw.c
+ compat/winansi.c
+ compat/win32/flush.c
+ compat/win32/path-utils.c
+ compat/win32/pthread.c
+ compat/win32mmap.c
+ compat/win32/syslog.c
+ compat/win32/trace2_win32_process_info.c
+ compat/win32/dirent.c
+ compat/nedmalloc/nedmalloc.c
+ compat/strdup.c)
set(NO_UNIX_SOCKETS 1)
elseif(CMAKE_SYSTEM_NAME STREQUAL "Linux")
diff --git a/environment.c b/environment.c
index fd0501e77a5..3e3620d759f 100644
--- a/environment.c
+++ b/environment.c
@@ -44,6 +44,7 @@ int zlib_compression_level = Z_BEST_SPEED;
int pack_compression_level = Z_DEFAULT_COMPRESSION;
int fsync_object_files;
int use_fsync = -1;
+enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT;
size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE;
size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT;
size_t delta_base_cache_limit = 96 * 1024 * 1024;
diff --git a/git-compat-util.h b/git-compat-util.h
index d210cff058c..00356476a9d 100644
--- a/git-compat-util.h
+++ b/git-compat-util.h
@@ -1271,6 +1271,30 @@ __attribute__((format (printf, 1, 2))) NORETURN
void BUG(const char *fmt, ...);
#endif
+#ifdef __APPLE__
+#define FSYNC_METHOD_DEFAULT FSYNC_METHOD_WRITEOUT_ONLY
+#else
+#define FSYNC_METHOD_DEFAULT FSYNC_METHOD_FSYNC
+#endif
+
+enum fsync_action {
+ FSYNC_WRITEOUT_ONLY,
+ FSYNC_HARDWARE_FLUSH
+};
+
+/*
+ * Issues an fsync against the specified file according to the specified mode.
+ *
+ * FSYNC_WRITEOUT_ONLY attempts to use interfaces available on some operating
+ * systems to flush the OS cache without issuing a flush command to the storage
+ * controller. If those interfaces are unavailable, the function fails with
+ * ENOSYS.
+ *
+ * FSYNC_HARDWARE_FLUSH does an OS writeout and hardware flush to ensure that
+ * changes are durable. It is not expected to fail.
+ */
+int git_fsync(int fd, enum fsync_action action);
+
/*
* Preserves errno, prints a message, but gives no warning for ENOENT.
* Returns 0 on success, which includes trying to unlink an object that does
diff --git a/wrapper.c b/wrapper.c
index 1108e4840a4..354d784c034 100644
--- a/wrapper.c
+++ b/wrapper.c
@@ -546,6 +546,70 @@ int xmkstemp_mode(char *filename_template, int mode)
return fd;
}
+/*
+ * Some platforms return EINTR from fsync. Since fsync is invoked in some
+ * cases by a wrapper that dies on failure, do not expose EINTR to callers.
+ */
+static int fsync_loop(int fd)
+{
+ int err;
+
+ do {
+ err = fsync(fd);
+ } while (err < 0 && errno == EINTR);
+ return err;
+}
+
+int git_fsync(int fd, enum fsync_action action)
+{
+ switch (action) {
+ case FSYNC_WRITEOUT_ONLY:
+
+#ifdef __APPLE__
+ /*
+ * On macOS, fsync just causes filesystem cache writeback but
+ * does not flush hardware caches.
+ */
+ return fsync_loop(fd);
+#endif
+
+#ifdef HAVE_SYNC_FILE_RANGE
+ /*
+ * On linux 2.6.17 and above, sync_file_range is the way to
+ * issue a writeback without a hardware flush. An offset of
+ * 0 and size of 0 indicates writeout of the entire file and the
+ * wait flags ensure that all dirty data is written to the disk
+ * (potentially in a disk-side cache) before we continue.
+ */
+
+ return sync_file_range(fd, 0, 0, SYNC_FILE_RANGE_WAIT_BEFORE |
+ SYNC_FILE_RANGE_WRITE |
+ SYNC_FILE_RANGE_WAIT_AFTER);
+#endif
+
+#ifdef fsync_no_flush
+ return fsync_no_flush(fd);
+#endif
+
+ errno = ENOSYS;
+ return -1;
+
+ case FSYNC_HARDWARE_FLUSH:
+ /*
+ * On macOS, a special fcntl is required to really flush the
+ * caches within the storage controller. As of this writing,
+ * this is a very expensive operation on Apple SSDs.
+ */
+#ifdef __APPLE__
+ return fcntl(fd, F_FULLFSYNC);
+#else
+ return fsync_loop(fd);
+#endif
+ default:
+ BUG("unexpected git_fsync(%d) call", action);
+ }
+}
+
static int warn_if_unremovable(const char *op, const char *file, int rc)
{
int err;
diff --git a/write-or-die.c b/write-or-die.c
index a3d5784cec9..9faa5f9f563 100644
--- a/write-or-die.c
+++ b/write-or-die.c
@@ -62,10 +62,13 @@ void fsync_or_die(int fd, const char *msg)
use_fsync = git_env_bool("GIT_TEST_FSYNC", 1);
if (!use_fsync)
return;
- while (fsync(fd) < 0) {
- if (errno != EINTR)
- die_errno("fsync error on '%s'", msg);
- }
+
+ if (fsync_method == FSYNC_METHOD_WRITEOUT_ONLY &&
+ git_fsync(fd, FSYNC_WRITEOUT_ONLY) >= 0)
+ return;
+
+ if (git_fsync(fd, FSYNC_HARDWARE_FLUSH) < 0)
+ die_errno("fsync error on '%s'", msg);
}
void write_or_die(int fd, const void *buf, size_t count)
--
gitgitgadget
^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v6 3/6] core.fsync: introduce granular fsync control infrastructure
2022-03-10 22:43 ` [PATCH v6 0/6] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget
2022-03-10 22:43 ` [PATCH v6 1/6] wrapper: make inclusion of Windows csprng header tightly scoped Neeraj Singh via GitGitGadget
2022-03-10 22:43 ` [PATCH v6 2/6] core.fsyncmethod: add writeout-only mode Neeraj Singh via GitGitGadget
@ 2022-03-10 22:43 ` Neeraj Singh via GitGitGadget
2022-03-10 22:43 ` [PATCH v6 4/6] core.fsync: add configuration parsing Neeraj Singh via GitGitGadget
` (4 subsequent siblings)
7 siblings, 0 replies; 122+ messages in thread
From: Neeraj Singh via GitGitGadget @ 2022-03-10 22:43 UTC (permalink / raw)
To: git
Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps, sandals,
Johannes Schindelin, Neeraj K. Singh, Neeraj Singh
From: Neeraj Singh <neerajsi@microsoft.com>
This commit introduces the infrastructure for the core.fsync
configuration knob. The repository components we want to sync
are identified by flags so that we can turn on or off syncing
for specific components.
If core.fsyncObjectFiles is set and the core.fsync configuration
also includes FSYNC_COMPONENT_LOOSE_OBJECT, we will fsync any
loose objects. This picks the strictest data integrity behavior
if core.fsync and core.fsyncObjectFiles are set to conflicting values.
This change introduces the currently unused fsync_component
helper, which will be used by a later patch that adds fsyncing to
the refs backend.
Actual configuration and documentation of the fsync components
list are in other patches in the series to separate review of
the underlying mechanism from the policy of how it's configured.
Helped-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
---
builtin/fast-import.c | 2 +-
builtin/index-pack.c | 4 ++--
builtin/pack-objects.c | 24 +++++++++++++++++-------
bulk-checkin.c | 5 +++--
cache.h | 23 +++++++++++++++++++++++
commit-graph.c | 3 ++-
csum-file.c | 5 +++--
csum-file.h | 3 ++-
environment.c | 1 +
midx.c | 3 ++-
object-file.c | 13 +++++++++----
pack-bitmap-write.c | 3 ++-
pack-write.c | 13 +++++++------
read-cache.c | 2 +-
write-or-die.c | 26 ++++++++++++++++++++++----
15 files changed, 97 insertions(+), 33 deletions(-)
diff --git a/builtin/fast-import.c b/builtin/fast-import.c
index b7105fcad9b..f2c036a8955 100644
--- a/builtin/fast-import.c
+++ b/builtin/fast-import.c
@@ -865,7 +865,7 @@ static void end_packfile(void)
struct tag *t;
close_pack_windows(pack_data);
- finalize_hashfile(pack_file, cur_pack_oid.hash, 0);
+ finalize_hashfile(pack_file, cur_pack_oid.hash, FSYNC_COMPONENT_PACK, 0);
fixup_pack_header_footer(pack_data->pack_fd, pack_data->hash,
pack_data->pack_name, object_count,
cur_pack_oid.hash, pack_size);
diff --git a/builtin/index-pack.c b/builtin/index-pack.c
index c45273de3b1..c5f12f14df5 100644
--- a/builtin/index-pack.c
+++ b/builtin/index-pack.c
@@ -1290,7 +1290,7 @@ static void conclude_pack(int fix_thin_pack, const char *curr_pack, unsigned cha
nr_objects - nr_objects_initial);
stop_progress_msg(&progress, msg.buf);
strbuf_release(&msg);
- finalize_hashfile(f, tail_hash, 0);
+ finalize_hashfile(f, tail_hash, FSYNC_COMPONENT_PACK, 0);
hashcpy(read_hash, pack_hash);
fixup_pack_header_footer(output_fd, pack_hash,
curr_pack, nr_objects,
@@ -1512,7 +1512,7 @@ static void final(const char *final_pack_name, const char *curr_pack_name,
if (!from_stdin) {
close(input_fd);
} else {
- fsync_or_die(output_fd, curr_pack_name);
+ fsync_component_or_die(FSYNC_COMPONENT_PACK, output_fd, curr_pack_name);
err = close(output_fd);
if (err)
die_errno(_("error while closing pack file"));
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index 178e611f09d..c14fee8e99f 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -1199,16 +1199,26 @@ static void write_pack_file(void)
display_progress(progress_state, written);
}
- /*
- * Did we write the wrong # entries in the header?
- * If so, rewrite it like in fast-import
- */
if (pack_to_stdout) {
- finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_CLOSE);
+ /*
+ * We never fsync when writing to stdout since we may
+ * not be writing to an actual pack file. For instance,
+ * the upload-pack code passes a pipe here. Calling
+ * fsync on a pipe results in unnecessary
+ * synchronization with the reader on some platforms.
+ */
+ finalize_hashfile(f, hash, FSYNC_COMPONENT_NONE,
+ CSUM_HASH_IN_STREAM | CSUM_CLOSE);
} else if (nr_written == nr_remaining) {
- finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
+ finalize_hashfile(f, hash, FSYNC_COMPONENT_PACK,
+ CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
} else {
- int fd = finalize_hashfile(f, hash, 0);
+ /*
+ * If we wrote the wrong number of entries in the
+ * header, rewrite it like in fast-import.
+ */
+
+ int fd = finalize_hashfile(f, hash, FSYNC_COMPONENT_PACK, 0);
fixup_pack_header_footer(fd, hash, pack_tmp_name,
nr_written, hash, offset);
close(fd);
diff --git a/bulk-checkin.c b/bulk-checkin.c
index 8785b2ac806..a2cf9dcbc8d 100644
--- a/bulk-checkin.c
+++ b/bulk-checkin.c
@@ -53,9 +53,10 @@ static void finish_bulk_checkin(struct bulk_checkin_state *state)
unlink(state->pack_tmp_name);
goto clear_exit;
} else if (state->nr_written == 1) {
- finalize_hashfile(state->f, hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
+ finalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK,
+ CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
} else {
- int fd = finalize_hashfile(state->f, hash, 0);
+ int fd = finalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK, 0);
fixup_pack_header_footer(fd, hash, state->pack_tmp_name,
state->nr_written, hash,
state->offset);
diff --git a/cache.h b/cache.h
index 82f0194a3dd..7ac1959258d 100644
--- a/cache.h
+++ b/cache.h
@@ -993,6 +993,27 @@ void reset_shared_repository(void);
extern int read_replace_refs;
extern char *git_replace_ref_base;
+/*
+ * These values are used to help identify parts of a repository to fsync.
+ * FSYNC_COMPONENT_NONE identifies data that will not be a persistent part of the
+ * repository and so shouldn't be fsynced.
+ */
+enum fsync_component {
+ FSYNC_COMPONENT_NONE,
+ FSYNC_COMPONENT_LOOSE_OBJECT = 1 << 0,
+ FSYNC_COMPONENT_PACK = 1 << 1,
+ FSYNC_COMPONENT_PACK_METADATA = 1 << 2,
+ FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3,
+};
+
+#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \
+ FSYNC_COMPONENT_PACK_METADATA | \
+ FSYNC_COMPONENT_COMMIT_GRAPH)
+
+/*
+ * A bitmask indicating which components of the repo should be fsynced.
+ */
+extern enum fsync_component fsync_components;
extern int fsync_object_files;
extern int use_fsync;
@@ -1707,6 +1728,8 @@ int copy_file_with_time(const char *dst, const char *src, int mode);
void write_or_die(int fd, const void *buf, size_t count);
void fsync_or_die(int fd, const char *);
+int fsync_component(enum fsync_component component, int fd);
+void fsync_component_or_die(enum fsync_component component, int fd, const char *msg);
ssize_t read_in_full(int fd, void *buf, size_t count);
ssize_t write_in_full(int fd, const void *buf, size_t count);
diff --git a/commit-graph.c b/commit-graph.c
index 265c010122e..64897f57d9f 100644
--- a/commit-graph.c
+++ b/commit-graph.c
@@ -1942,7 +1942,8 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)
}
close_commit_graph(ctx->r->objects);
- finalize_hashfile(f, file_hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC);
+ finalize_hashfile(f, file_hash, FSYNC_COMPONENT_COMMIT_GRAPH,
+ CSUM_HASH_IN_STREAM | CSUM_FSYNC);
free_chunkfile(cf);
if (ctx->split) {
diff --git a/csum-file.c b/csum-file.c
index 26e8a6df44e..59ef3398ca2 100644
--- a/csum-file.c
+++ b/csum-file.c
@@ -58,7 +58,8 @@ static void free_hashfile(struct hashfile *f)
free(f);
}
-int finalize_hashfile(struct hashfile *f, unsigned char *result, unsigned int flags)
+int finalize_hashfile(struct hashfile *f, unsigned char *result,
+ enum fsync_component component, unsigned int flags)
{
int fd;
@@ -69,7 +70,7 @@ int finalize_hashfile(struct hashfile *f, unsigned char *result, unsigned int fl
if (flags & CSUM_HASH_IN_STREAM)
flush(f, f->buffer, the_hash_algo->rawsz);
if (flags & CSUM_FSYNC)
- fsync_or_die(f->fd, f->name);
+ fsync_component_or_die(component, f->fd, f->name);
if (flags & CSUM_CLOSE) {
if (close(f->fd))
die_errno("%s: sha1 file error on close", f->name);
diff --git a/csum-file.h b/csum-file.h
index 291215b34eb..0d29f528fbc 100644
--- a/csum-file.h
+++ b/csum-file.h
@@ -1,6 +1,7 @@
#ifndef CSUM_FILE_H
#define CSUM_FILE_H
+#include "cache.h"
#include "hash.h"
struct progress;
@@ -38,7 +39,7 @@ int hashfile_truncate(struct hashfile *, struct hashfile_checkpoint *);
struct hashfile *hashfd(int fd, const char *name);
struct hashfile *hashfd_check(const char *name);
struct hashfile *hashfd_throughput(int fd, const char *name, struct progress *tp);
-int finalize_hashfile(struct hashfile *, unsigned char *, unsigned int);
+int finalize_hashfile(struct hashfile *, unsigned char *, enum fsync_component, unsigned int);
void hashwrite(struct hashfile *, const void *, unsigned int);
void hashflush(struct hashfile *f);
void crc32_begin(struct hashfile *);
diff --git a/environment.c b/environment.c
index 3e3620d759f..36ca5fb2e77 100644
--- a/environment.c
+++ b/environment.c
@@ -45,6 +45,7 @@ int pack_compression_level = Z_DEFAULT_COMPRESSION;
int fsync_object_files;
int use_fsync = -1;
enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT;
+enum fsync_component fsync_components = FSYNC_COMPONENTS_DEFAULT;
size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE;
size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT;
size_t delta_base_cache_limit = 96 * 1024 * 1024;
diff --git a/midx.c b/midx.c
index 865170bad05..107365d2114 100644
--- a/midx.c
+++ b/midx.c
@@ -1438,7 +1438,8 @@ static int write_midx_internal(const char *object_dir,
write_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs);
write_chunkfile(cf, &ctx);
- finalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM);
+ finalize_hashfile(f, midx_hash, FSYNC_COMPONENT_PACK_METADATA,
+ CSUM_FSYNC | CSUM_HASH_IN_STREAM);
free_chunkfile(cf);
if (flags & MIDX_WRITE_REV_INDEX &&
diff --git a/object-file.c b/object-file.c
index 03bd6a3baf3..e3f0bf27ff1 100644
--- a/object-file.c
+++ b/object-file.c
@@ -1849,11 +1849,16 @@ int hash_object_file(const struct git_hash_algo *algo, const void *buf,
/* Finalize a file on disk, and close it. */
static void close_loose_object(int fd)
{
- if (!the_repository->objects->odb->will_destroy) {
- if (fsync_object_files)
- fsync_or_die(fd, "loose object file");
- }
+ if (the_repository->objects->odb->will_destroy)
+ goto out;
+ if (fsync_object_files > 0)
+ fsync_or_die(fd, "loose object file");
+ else
+ fsync_component_or_die(FSYNC_COMPONENT_LOOSE_OBJECT, fd,
+ "loose object file");
+
+out:
if (close(fd) != 0)
die_errno(_("error when closing loose object file"));
}
diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c
index cab3eaa2acd..cf681547f2e 100644
--- a/pack-bitmap-write.c
+++ b/pack-bitmap-write.c
@@ -719,7 +719,8 @@ void bitmap_writer_finish(struct pack_idx_entry **index,
if (options & BITMAP_OPT_HASH_CACHE)
write_hash_cache(f, index, index_nr);
- finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
+ finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA,
+ CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);
if (adjust_shared_perm(tmp_file.buf))
die_errno("unable to make temporary bitmap file readable");
diff --git a/pack-write.c b/pack-write.c
index a5846f3a346..51812cb1299 100644
--- a/pack-write.c
+++ b/pack-write.c
@@ -159,9 +159,9 @@ const char *write_idx_file(const char *index_name, struct pack_idx_entry **objec
}
hashwrite(f, sha1, the_hash_algo->rawsz);
- finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE |
- ((opts->flags & WRITE_IDX_VERIFY)
- ? 0 : CSUM_FSYNC));
+ finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA,
+ CSUM_HASH_IN_STREAM | CSUM_CLOSE |
+ ((opts->flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC));
return index_name;
}
@@ -281,8 +281,9 @@ const char *write_rev_file_order(const char *rev_name,
if (rev_name && adjust_shared_perm(rev_name) < 0)
die(_("failed to make %s readable"), rev_name);
- finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE |
- ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC));
+ finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA,
+ CSUM_HASH_IN_STREAM | CSUM_CLOSE |
+ ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC));
return rev_name;
}
@@ -390,7 +391,7 @@ void fixup_pack_header_footer(int pack_fd,
the_hash_algo->final_fn(partial_pack_hash, &old_hash_ctx);
the_hash_algo->final_fn(new_pack_hash, &new_hash_ctx);
write_or_die(pack_fd, new_pack_hash, the_hash_algo->rawsz);
- fsync_or_die(pack_fd, pack_name);
+ fsync_component_or_die(FSYNC_COMPONENT_PACK, pack_fd, pack_name);
}
char *index_pack_lockfile(int ip_out, int *is_well_formed)
diff --git a/read-cache.c b/read-cache.c
index 79b9b99ebf7..df869691fd4 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -3089,7 +3089,7 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile,
return -1;
}
- finalize_hashfile(f, istate->oid.hash, CSUM_HASH_IN_STREAM);
+ finalize_hashfile(f, istate->oid.hash, FSYNC_COMPONENT_NONE, CSUM_HASH_IN_STREAM);
if (close_tempfile_gently(tempfile)) {
error(_("could not close '%s'"), get_tempfile_path(tempfile));
return -1;
diff --git a/write-or-die.c b/write-or-die.c
index 9faa5f9f563..c4fd91b5b43 100644
--- a/write-or-die.c
+++ b/write-or-die.c
@@ -56,21 +56,39 @@ void fprintf_or_die(FILE *f, const char *fmt, ...)
}
}
-void fsync_or_die(int fd, const char *msg)
+static int maybe_fsync(int fd)
{
if (use_fsync < 0)
use_fsync = git_env_bool("GIT_TEST_FSYNC", 1);
if (!use_fsync)
- return;
+ return 0;
if (fsync_method == FSYNC_METHOD_WRITEOUT_ONLY &&
git_fsync(fd, FSYNC_WRITEOUT_ONLY) >= 0)
- return;
+ return 0;
+
+ return git_fsync(fd, FSYNC_HARDWARE_FLUSH);
+}
- if (git_fsync(fd, FSYNC_HARDWARE_FLUSH) < 0)
+void fsync_or_die(int fd, const char *msg)
+{
+ if (maybe_fsync(fd) < 0)
die_errno("fsync error on '%s'", msg);
}
+int fsync_component(enum fsync_component component, int fd)
+{
+ if (fsync_components & component)
+ return maybe_fsync(fd);
+ return 0;
+}
+
+void fsync_component_or_die(enum fsync_component component, int fd, const char *msg)
+{
+ if (fsync_components & component)
+ fsync_or_die(fd, msg);
+}
+
void write_or_die(int fd, const void *buf, size_t count)
{
if (write_in_full(fd, buf, count) < 0) {
--
gitgitgadget
^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v6 4/6] core.fsync: add configuration parsing
2022-03-10 22:43 ` [PATCH v6 0/6] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget
` (2 preceding siblings ...)
2022-03-10 22:43 ` [PATCH v6 3/6] core.fsync: introduce granular fsync control infrastructure Neeraj Singh via GitGitGadget
@ 2022-03-10 22:43 ` Neeraj Singh via GitGitGadget
2022-03-28 11:06 ` Jiang Xin
2022-03-10 22:43 ` [PATCH v6 5/6] core.fsync: new option to harden the index Neeraj Singh via GitGitGadget
` (3 subsequent siblings)
7 siblings, 1 reply; 122+ messages in thread
From: Neeraj Singh via GitGitGadget @ 2022-03-10 22:43 UTC (permalink / raw)
To: git
Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps, sandals,
Johannes Schindelin, Neeraj K. Singh, Neeraj Singh
From: Neeraj Singh <neerajsi@microsoft.com>
This change introduces code to parse the core.fsync setting and
configure the fsync_components variable.
core.fsync is configured as a comma-separated list of component names to
sync. Each time a core.fsync variable is encountered in the
configuration heirarchy, we start off with a clean state with the
platform default value. Passing 'none' resets the value to indicate
nothing will be synced. We gather all negative and positive entries from
the comma separated list and then compute the new value by removing all
the negative entries and adding all of the positive entries.
We issue a warning for components that are not recognized so that the
configuration code is compatible with configs from future versions of
Git with more repo components.
Complete documentation for the new setting is included in a later patch
in the series so that it can be reviewed once in final form.
Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
---
Documentation/config/core.txt | 9 +++--
config.c | 76 +++++++++++++++++++++++++++++++++++
environment.c | 2 +-
3 files changed, 82 insertions(+), 5 deletions(-)
diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
index dbb134f7136..ab911d6e269 100644
--- a/Documentation/config/core.txt
+++ b/Documentation/config/core.txt
@@ -558,11 +558,12 @@ core.fsyncMethod::
core.fsyncObjectFiles::
This boolean will enable 'fsync()' when writing object files.
+ This setting is deprecated. Use core.fsync instead.
+
-This is a total waste of time and effort on a filesystem that orders
-data writes properly, but can be useful for filesystems that do not use
-journalling (traditional UNIX filesystems) or that only journal metadata
-and not file contents (OS X's HFS+, or Linux ext3 with "data=writeback").
+This setting affects data added to the Git repository in loose-object
+form. When set to true, Git will issue an fsync or similar system call
+to flush caches so that loose-objects remain consistent in the face
+of a unclean system shutdown.
core.preloadIndex::
Enable parallel index preload for operations like 'git diff'
diff --git a/config.c b/config.c
index f3ff80b01c9..94a4598b5fc 100644
--- a/config.c
+++ b/config.c
@@ -1323,6 +1323,73 @@ static int git_parse_maybe_bool_text(const char *value)
return -1;
}
+static const struct fsync_component_name {
+ const char *name;
+ enum fsync_component component_bits;
+} fsync_component_names[] = {
+ { "loose-object", FSYNC_COMPONENT_LOOSE_OBJECT },
+ { "pack", FSYNC_COMPONENT_PACK },
+ { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA },
+ { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH },
+};
+
+static enum fsync_component parse_fsync_components(const char *var, const char *string)
+{
+ enum fsync_component current = FSYNC_COMPONENTS_DEFAULT;
+ enum fsync_component positive = 0, negative = 0;
+
+ while (string) {
+ int i;
+ size_t len;
+ const char *ep;
+ int negated = 0;
+ int found = 0;
+
+ string = string + strspn(string, ", \t\n\r");
+ ep = strchrnul(string, ',');
+ len = ep - string;
+ if (!strcmp(string, "none")) {
+ current = FSYNC_COMPONENT_NONE;
+ goto next_name;
+ }
+
+ if (*string == '-') {
+ negated = 1;
+ string++;
+ len--;
+ if (!len)
+ warning(_("invalid value for variable %s"), var);
+ }
+
+ if (!len)
+ break;
+
+ for (i = 0; i < ARRAY_SIZE(fsync_component_names); ++i) {
+ const struct fsync_component_name *n = &fsync_component_names[i];
+
+ if (strncmp(n->name, string, len))
+ continue;
+
+ found = 1;
+ if (negated)
+ negative |= n->component_bits;
+ else
+ positive |= n->component_bits;
+ }
+
+ if (!found) {
+ char *component = xstrndup(string, len);
+ warning(_("ignoring unknown core.fsync component '%s'"), component);
+ free(component);
+ }
+
+next_name:
+ string = ep;
+ }
+
+ return (current & ~negative) | positive;
+}
+
int git_parse_maybe_bool(const char *value)
{
int v = git_parse_maybe_bool_text(value);
@@ -1600,6 +1667,13 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
return 0;
}
+ if (!strcmp(var, "core.fsync")) {
+ if (!value)
+ return config_error_nonbool(var);
+ fsync_components = parse_fsync_components(var, value);
+ return 0;
+ }
+
if (!strcmp(var, "core.fsyncmethod")) {
if (!value)
return config_error_nonbool(var);
@@ -1613,6 +1687,8 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
}
if (!strcmp(var, "core.fsyncobjectfiles")) {
+ if (fsync_object_files < 0)
+ warning(_("core.fsyncobjectfiles is deprecated; use core.fsync instead"));
fsync_object_files = git_config_bool(var, value);
return 0;
}
diff --git a/environment.c b/environment.c
index 36ca5fb2e77..698f03a2f47 100644
--- a/environment.c
+++ b/environment.c
@@ -42,7 +42,7 @@ const char *git_attributes_file;
const char *git_hooks_path;
int zlib_compression_level = Z_BEST_SPEED;
int pack_compression_level = Z_DEFAULT_COMPRESSION;
-int fsync_object_files;
+int fsync_object_files = -1;
int use_fsync = -1;
enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT;
enum fsync_component fsync_components = FSYNC_COMPONENTS_DEFAULT;
--
gitgitgadget
^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v6 5/6] core.fsync: new option to harden the index
2022-03-10 22:43 ` [PATCH v6 0/6] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget
` (3 preceding siblings ...)
2022-03-10 22:43 ` [PATCH v6 4/6] core.fsync: add configuration parsing Neeraj Singh via GitGitGadget
@ 2022-03-10 22:43 ` Neeraj Singh via GitGitGadget
2022-03-10 22:43 ` [PATCH v6 6/6] core.fsync: documentation and user-friendly aggregate options Neeraj Singh via GitGitGadget
` (2 subsequent siblings)
7 siblings, 0 replies; 122+ messages in thread
From: Neeraj Singh via GitGitGadget @ 2022-03-10 22:43 UTC (permalink / raw)
To: git
Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps, sandals,
Johannes Schindelin, Neeraj K. Singh, Neeraj Singh
From: Neeraj Singh <neerajsi@microsoft.com>
This commit introduces the new ability for the user to harden
the index. In the event of a system crash, the index must be
durable for the user to actually find a file that has been added
to the repo and then deleted from the working tree.
We use the presence of the COMMIT_LOCK flag and absence of the
alternate_index_output as a proxy for determining whether we're
updating the persistent index of the repo or some temporary
index. We don't sync these temporary indexes.
Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
---
cache.h | 1 +
config.c | 1 +
read-cache.c | 19 +++++++++++++------
3 files changed, 15 insertions(+), 6 deletions(-)
diff --git a/cache.h b/cache.h
index 7ac1959258d..e08eeac6c15 100644
--- a/cache.h
+++ b/cache.h
@@ -1004,6 +1004,7 @@ enum fsync_component {
FSYNC_COMPONENT_PACK = 1 << 1,
FSYNC_COMPONENT_PACK_METADATA = 1 << 2,
FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3,
+ FSYNC_COMPONENT_INDEX = 1 << 4,
};
#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \
diff --git a/config.c b/config.c
index 94a4598b5fc..80f33c91982 100644
--- a/config.c
+++ b/config.c
@@ -1331,6 +1331,7 @@ static const struct fsync_component_name {
{ "pack", FSYNC_COMPONENT_PACK },
{ "pack-metadata", FSYNC_COMPONENT_PACK_METADATA },
{ "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH },
+ { "index", FSYNC_COMPONENT_INDEX },
};
static enum fsync_component parse_fsync_components(const char *var, const char *string)
diff --git a/read-cache.c b/read-cache.c
index df869691fd4..7683b679258 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -2842,7 +2842,7 @@ static int record_ieot(void)
* rely on it.
*/
static int do_write_index(struct index_state *istate, struct tempfile *tempfile,
- int strip_extensions)
+ int strip_extensions, unsigned flags)
{
uint64_t start = getnanotime();
struct hashfile *f;
@@ -2856,6 +2856,7 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile,
struct strbuf previous_name_buf = STRBUF_INIT, *previous_name;
int drop_cache_tree = istate->drop_cache_tree;
off_t offset;
+ int csum_fsync_flag;
int ieot_entries = 1;
struct index_entry_offset_table *ieot = NULL;
int nr, nr_threads;
@@ -3089,7 +3090,13 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile,
return -1;
}
- finalize_hashfile(f, istate->oid.hash, FSYNC_COMPONENT_NONE, CSUM_HASH_IN_STREAM);
+ csum_fsync_flag = 0;
+ if (!alternate_index_output && (flags & COMMIT_LOCK))
+ csum_fsync_flag = CSUM_FSYNC;
+
+ finalize_hashfile(f, istate->oid.hash, FSYNC_COMPONENT_INDEX,
+ CSUM_HASH_IN_STREAM | csum_fsync_flag);
+
if (close_tempfile_gently(tempfile)) {
error(_("could not close '%s'"), get_tempfile_path(tempfile));
return -1;
@@ -3144,7 +3151,7 @@ static int do_write_locked_index(struct index_state *istate, struct lock_file *l
*/
trace2_region_enter_printf("index", "do_write_index", the_repository,
"%s", get_lock_file_path(lock));
- ret = do_write_index(istate, lock->tempfile, 0);
+ ret = do_write_index(istate, lock->tempfile, 0, flags);
trace2_region_leave_printf("index", "do_write_index", the_repository,
"%s", get_lock_file_path(lock));
@@ -3238,7 +3245,7 @@ static int clean_shared_index_files(const char *current_hex)
}
static int write_shared_index(struct index_state *istate,
- struct tempfile **temp)
+ struct tempfile **temp, unsigned flags)
{
struct split_index *si = istate->split_index;
int ret, was_full = !istate->sparse_index;
@@ -3248,7 +3255,7 @@ static int write_shared_index(struct index_state *istate,
trace2_region_enter_printf("index", "shared/do_write_index",
the_repository, "%s", get_tempfile_path(*temp));
- ret = do_write_index(si->base, *temp, 1);
+ ret = do_write_index(si->base, *temp, 1, flags);
trace2_region_leave_printf("index", "shared/do_write_index",
the_repository, "%s", get_tempfile_path(*temp));
@@ -3357,7 +3364,7 @@ int write_locked_index(struct index_state *istate, struct lock_file *lock,
ret = do_write_locked_index(istate, lock, flags);
goto out;
}
- ret = write_shared_index(istate, &temp);
+ ret = write_shared_index(istate, &temp, flags);
saved_errno = errno;
if (is_tempfile_active(temp))
--
gitgitgadget
^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v6 6/6] core.fsync: documentation and user-friendly aggregate options
2022-03-10 22:43 ` [PATCH v6 0/6] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget
` (4 preceding siblings ...)
2022-03-10 22:43 ` [PATCH v6 5/6] core.fsync: new option to harden the index Neeraj Singh via GitGitGadget
@ 2022-03-10 22:43 ` Neeraj Singh via GitGitGadget
2022-03-15 19:12 ` [PATCH v7] " Neeraj Singh
2022-03-10 23:34 ` [PATCH v6 0/6] A design for future-proofing fsync() configuration Junio C Hamano
2022-03-11 9:58 ` [PATCH v2] core.fsync: new option to harden references Patrick Steinhardt
7 siblings, 1 reply; 122+ messages in thread
From: Neeraj Singh via GitGitGadget @ 2022-03-10 22:43 UTC (permalink / raw)
To: git
Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps, sandals,
Johannes Schindelin, Neeraj K. Singh, Neeraj Singh
From: Neeraj Singh <neerajsi@microsoft.com>
This commit adds aggregate options for the core.fsync setting that are
more user-friendly. These options are specified in terms of 'levels of
safety', indicating which Git operations are considered to be sync
points for durability.
The new documentation is also included here in its entirety for ease of
review.
Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
---
Documentation/config/core.txt | 40 +++++++++++++++++++++++++++++++++++
cache.h | 23 +++++++++++++++++---
config.c | 5 +++++
3 files changed, 65 insertions(+), 3 deletions(-)
diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
index ab911d6e269..37105a7be40 100644
--- a/Documentation/config/core.txt
+++ b/Documentation/config/core.txt
@@ -547,6 +547,46 @@ core.whitespace::
is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent`
errors. The default tab width is 8. Allowed values are 1 to 63.
+core.fsync::
+ A comma-separated list of components of the repository that
+ should be hardened via the core.fsyncMethod when created or
+ modified. You can disable hardening of any component by
+ prefixing it with a '-'. Items that are not hardened may be
+ lost in the event of an unclean system shutdown. Unless you
+ have special requirements, it is recommended that you leave
+ this option empty or pick one of `committed`, `added`,
+ or `all`.
++
+When this configuration is encountered, the set of components starts with
+the platform default value, disabled components are removed, and additional
+components are added. `none` resets the state so that the platform default
+is ignored.
++
+The empty string resets the fsync configuration to the platform
+default. The platform default on most platform is equivalent to
+`core.fsync=committed,-loose-object`, which has good performance,
+but risks losing recent work in the event of an unclean system shutdown.
++
+* `none` clears the set of fsynced components.
+* `loose-object` hardens objects added to the repo in loose-object form.
+* `pack` hardens objects added to the repo in packfile form.
+* `pack-metadata` hardens packfile bitmaps and indexes.
+* `commit-graph` hardens the commit graph file.
+* `index` hardens the index when it is modified.
+* `objects` is an aggregate option that is equivalent to
+ `loose-object,pack`.
+* `derived-metadata` is an aggregate option that is equivalent to
+ `pack-metadata,commit-graph`.
+* `committed` is an aggregate option that is currently equivalent to
+ `objects`. This mode sacrifices some performance to ensure that work
+ that is committed to the repository with `git commit` or similar commands
+ is hardened.
+* `added` is an aggregate option that is currently equivalent to
+ `committed,index`. This mode sacrifices additional performance to
+ ensure that the results of commands like `git add` and similar operations
+ are hardened.
+* `all` is an aggregate option that syncs all individual components above.
+
core.fsyncMethod::
A value indicating the strategy Git will use to harden repository data
using fsync and related primitives.
diff --git a/cache.h b/cache.h
index e08eeac6c15..86680f144ec 100644
--- a/cache.h
+++ b/cache.h
@@ -1007,9 +1007,26 @@ enum fsync_component {
FSYNC_COMPONENT_INDEX = 1 << 4,
};
-#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \
- FSYNC_COMPONENT_PACK_METADATA | \
- FSYNC_COMPONENT_COMMIT_GRAPH)
+#define FSYNC_COMPONENTS_OBJECTS (FSYNC_COMPONENT_LOOSE_OBJECT | \
+ FSYNC_COMPONENT_PACK)
+
+#define FSYNC_COMPONENTS_DERIVED_METADATA (FSYNC_COMPONENT_PACK_METADATA | \
+ FSYNC_COMPONENT_COMMIT_GRAPH)
+
+#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENTS_OBJECTS | \
+ FSYNC_COMPONENTS_DERIVED_METADATA | \
+ ~FSYNC_COMPONENT_LOOSE_OBJECT)
+
+#define FSYNC_COMPONENTS_COMMITTED (FSYNC_COMPONENTS_OBJECTS)
+
+#define FSYNC_COMPONENTS_ADDED (FSYNC_COMPONENTS_COMMITTED | \
+ FSYNC_COMPONENT_INDEX)
+
+#define FSYNC_COMPONENTS_ALL (FSYNC_COMPONENT_LOOSE_OBJECT | \
+ FSYNC_COMPONENT_PACK | \
+ FSYNC_COMPONENT_PACK_METADATA | \
+ FSYNC_COMPONENT_COMMIT_GRAPH | \
+ FSYNC_COMPONENT_INDEX)
/*
* A bitmask indicating which components of the repo should be fsynced.
diff --git a/config.c b/config.c
index 80f33c91982..fd8e1659312 100644
--- a/config.c
+++ b/config.c
@@ -1332,6 +1332,11 @@ static const struct fsync_component_name {
{ "pack-metadata", FSYNC_COMPONENT_PACK_METADATA },
{ "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH },
{ "index", FSYNC_COMPONENT_INDEX },
+ { "objects", FSYNC_COMPONENTS_OBJECTS },
+ { "derived-metadata", FSYNC_COMPONENTS_DERIVED_METADATA },
+ { "committed", FSYNC_COMPONENTS_COMMITTED },
+ { "added", FSYNC_COMPONENTS_ADDED },
+ { "all", FSYNC_COMPONENTS_ALL },
};
static enum fsync_component parse_fsync_components(const char *var, const char *string)
--
gitgitgadget
^ permalink raw reply related [flat|nested] 122+ messages in thread
* Re: [PATCH 7/8] core.fsync: new option to harden loose references
2022-03-10 9:53 ` [PATCH 7/8] core.fsync: new option to harden loose references Patrick Steinhardt
2022-03-10 18:25 ` Junio C Hamano
@ 2022-03-10 22:54 ` Neeraj Singh
2022-03-11 6:40 ` Junio C Hamano
2 siblings, 0 replies; 122+ messages in thread
From: Neeraj Singh @ 2022-03-10 22:54 UTC (permalink / raw)
To: Patrick Steinhardt
Cc: Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren,
Ævar Arnfjörð Bjarmason, brian m. carlson,
Neeraj K. Singh, Junio C Hamano
On Thu, Mar 10, 2022 at 1:53 AM Patrick Steinhardt <ps@pks.im> wrote:
> diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
> index 973805e8a9..b67d3c340e 100644
> --- a/Documentation/config/core.txt
> +++ b/Documentation/config/core.txt
> @@ -564,8 +564,10 @@ core.fsync::
> * `pack-metadata` hardens packfile bitmaps and indexes.
> * `commit-graph` hardens the commit graph file.
> * `index` hardens the index when it is modified.
> +* `loose-ref` hardens references modified in the repo in loose-ref form.
> * `objects` is an aggregate option that is equivalent to
> `loose-object,pack`.
> +* `refs` is an aggregate option that is equivalent to `loose-ref`.
> * `derived-metadata` is an aggregate option that is equivalent to
> `pack-metadata,commit-graph`.
> * `default` is an aggregate option that is equivalent to
> diff --git a/cache.h b/cache.h
> index 63a95d1977..b56a56f539 100644
> --- a/cache.h
> +++ b/cache.h
> @@ -1005,11 +1005,14 @@ enum fsync_component {
> FSYNC_COMPONENT_PACK_METADATA = 1 << 2,
> FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3,
> FSYNC_COMPONENT_INDEX = 1 << 4,
> + FSYNC_COMPONENT_LOOSE_REF = 1 << 5,
> };
>
> #define FSYNC_COMPONENTS_OBJECTS (FSYNC_COMPONENT_LOOSE_OBJECT | \
> FSYNC_COMPONENT_PACK)
>
> +#define FSYNC_COMPONENTS_REFS (FSYNC_COMPONENT_LOOSE_REF)
> +
> #define FSYNC_COMPONENTS_DERIVED_METADATA (FSYNC_COMPONENT_PACK_METADATA | \
> FSYNC_COMPONENT_COMMIT_GRAPH)
>
> @@ -1026,7 +1029,8 @@ enum fsync_component {
> FSYNC_COMPONENT_PACK | \
> FSYNC_COMPONENT_PACK_METADATA | \
> FSYNC_COMPONENT_COMMIT_GRAPH | \
> - FSYNC_COMPONENT_INDEX)
> + FSYNC_COMPONENT_INDEX | \
> + FSYNC_COMPONENT_LOOSE_REF)
>
> /*
> * A bitmask indicating which components of the repo should be fsynced.
> diff --git a/config.c b/config.c
> index f03f27c3de..b5d3e6e404 100644
> --- a/config.c
> +++ b/config.c
> @@ -1332,7 +1332,9 @@ static const struct fsync_component_entry {
> { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA },
> { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH },
> { "index", FSYNC_COMPONENT_INDEX },
> + { "loose-ref", FSYNC_COMPONENT_LOOSE_REF },
> { "objects", FSYNC_COMPONENTS_OBJECTS },
> + { "refs", FSYNC_COMPONENTS_REFS },
> { "derived-metadata", FSYNC_COMPONENTS_DERIVED_METADATA },
> { "default", FSYNC_COMPONENTS_DEFAULT },
> { "committed", FSYNC_COMPONENTS_COMMITTED },
In terms of the 'preciousness-levels', refs should be included in
FSYNC_COMPONENTS_COMMITTED,
from which it will also be included in _ADDED.
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v6 0/6] A design for future-proofing fsync() configuration
2022-03-10 22:43 ` [PATCH v6 0/6] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget
` (5 preceding siblings ...)
2022-03-10 22:43 ` [PATCH v6 6/6] core.fsync: documentation and user-friendly aggregate options Neeraj Singh via GitGitGadget
@ 2022-03-10 23:34 ` Junio C Hamano
2022-03-11 0:03 ` Neeraj Singh
2022-03-13 23:50 ` Junio C Hamano
2022-03-11 9:58 ` [PATCH v2] core.fsync: new option to harden references Patrick Steinhardt
7 siblings, 2 replies; 122+ messages in thread
From: Junio C Hamano @ 2022-03-10 23:34 UTC (permalink / raw)
To: Neeraj K. Singh via GitGitGadget
Cc: git, rsbecker, bagasdotme, newren, avarab, nksingh85, ps, sandals,
Johannes Schindelin, Neeraj K. Singh
"Neeraj K. Singh via GitGitGadget" <gitgitgadget@gmail.com> writes:
> After this change, new persistent data files added to the repo will need to
> be added to the fsync_component enum and documented in the
> Documentation/config/core.txt text.
>
> V6 changes:
>
> * Move only the windows csprng includes into wrapper.c rather than all of
> them. This fixes the specific build issue due to broken Windows headers.
> [6]
> * Split the configuration parsing of core.fsync from the mechanism to focus
> the review.
> * Incorporate Patrick's patch at [7] into the core.fsync mechanism patch.
> * Pick the stricter one of core.fsyncObjectFiles and (fsync_components &
> FSYNC_COMPONENT_LOOSE_OBJECTS), to respect the older setting.
> * Issue a deprecation warning but keep parsing and honoring
> core.fsyncObjectFiles.
> * Change configuration parsing of core.fsync to always start with the
> platform default. none resets to the empty set. The comma separated list
> implies a set without regards to ordering now. This follows Junio's
> suggestion in [8].
> * Change the documentation of the core.fsync option to reflect the way the
> new parsing code works.
Hmph, this seems to make one test fail.
t5801-remote-helpers.sh (Wstat: 256 Tests: 31 Failed: 4)
Failed tests: 14-16, 31
Non-zero exit status: 1
Files=1, Tests=31, 2 wallclock secs ( 0.04 usr 0.00 sys + 1.40 cusr 1.62 csys = 3.06 CPU)
Result: FAIL
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v6 0/6] A design for future-proofing fsync() configuration
2022-03-10 23:34 ` [PATCH v6 0/6] A design for future-proofing fsync() configuration Junio C Hamano
@ 2022-03-11 0:03 ` Neeraj Singh
2022-03-11 18:50 ` Neeraj Singh
2022-03-13 23:50 ` Junio C Hamano
1 sibling, 1 reply; 122+ messages in thread
From: Neeraj Singh @ 2022-03-11 0:03 UTC (permalink / raw)
To: Junio C Hamano
Cc: Neeraj K. Singh via GitGitGadget, Git List, Randall S. Becker,
Bagas Sanjaya, Elijah Newren,
Ævar Arnfjörð Bjarmason, Patrick Steinhardt,
brian m. carlson, Johannes Schindelin, Neeraj K. Singh
On Thu, Mar 10, 2022 at 3:35 PM Junio C Hamano <gitster@pobox.com> wrote:
>
> "Neeraj K. Singh via GitGitGadget" <gitgitgadget@gmail.com> writes:
>
> > After this change, new persistent data files added to the repo will need to
> > be added to the fsync_component enum and documented in the
> > Documentation/config/core.txt text.
> >
> > V6 changes:
> >
> > * Move only the windows csprng includes into wrapper.c rather than all of
> > them. This fixes the specific build issue due to broken Windows headers.
> > [6]
> > * Split the configuration parsing of core.fsync from the mechanism to focus
> > the review.
> > * Incorporate Patrick's patch at [7] into the core.fsync mechanism patch.
> > * Pick the stricter one of core.fsyncObjectFiles and (fsync_components &
> > FSYNC_COMPONENT_LOOSE_OBJECTS), to respect the older setting.
> > * Issue a deprecation warning but keep parsing and honoring
> > core.fsyncObjectFiles.
> > * Change configuration parsing of core.fsync to always start with the
> > platform default. none resets to the empty set. The comma separated list
> > implies a set without regards to ordering now. This follows Junio's
> > suggestion in [8].
> > * Change the documentation of the core.fsync option to reflect the way the
> > new parsing code works.
>
> Hmph, this seems to make one test fail.
>
> t5801-remote-helpers.sh (Wstat: 256 Tests: 31 Failed: 4)
> Failed tests: 14-16, 31
> Non-zero exit status: 1
> Files=1, Tests=31, 2 wallclock secs ( 0.04 usr 0.00 sys + 1.40 cusr 1.62 csys = 3.06 CPU)
> Result: FAIL
Thanks for reporting this. I didn't see a failure in CI, nor when
running that specific test in mingw. I also munged my config to
include core.fsyncObjectFiles and didn't see a failure.
Could you please share some more verbose output of the test, so I can
look a bit deeper? In parallel, I'm trying again after merging my
changes onto seen.
Thanks,
Neeraj
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH 7/8] core.fsync: new option to harden loose references
2022-03-10 9:53 ` [PATCH 7/8] core.fsync: new option to harden loose references Patrick Steinhardt
2022-03-10 18:25 ` Junio C Hamano
2022-03-10 22:54 ` Neeraj Singh
@ 2022-03-11 6:40 ` Junio C Hamano
2022-03-11 9:15 ` Patrick Steinhardt
2 siblings, 1 reply; 122+ messages in thread
From: Junio C Hamano @ 2022-03-11 6:40 UTC (permalink / raw)
To: Patrick Steinhardt
Cc: git, rsbecker, bagasdotme, newren, avarab, nksingh85, sandals,
Neeraj K. Singh
Patrick Steinhardt <ps@pks.im> writes:
> @@ -1504,6 +1513,7 @@ static int files_copy_or_rename_ref(struct ref_store *ref_store,
> oidcpy(&lock->old_oid, &orig_oid);
>
> if (write_ref_to_lockfile(lock, &orig_oid, 0, &err) ||
> + files_sync_loose_ref(lock, &err) ||
> commit_ref_update(refs, lock, &orig_oid, logmsg, &err)) {
> error("unable to write current sha1 into %s: %s", newrefname, err.buf);
> strbuf_release(&err);
Given that write_ref_to_lockfile() on the success code path does this:
fd = get_lock_file_fd(&lock->lk);
if (write_in_full(fd, oid_to_hex(oid), the_hash_algo->hexsz) < 0 ||
write_in_full(fd, &term, 1) < 0 ||
close_ref_gently(lock) < 0) {
strbuf_addf(err,
"couldn't write '%s'", get_lock_file_path(&lock->lk));
unlock_ref(lock);
return -1;
}
return 0;
the above unfortunately does not work. By the time the new call to
files_sync_loose_ref() is made, lock->fd is closed by the call to
close_lock_file_gently() made in close_ref_gently(), and because of
that, you'll get an error like this:
Writing objects: 100% (3/3), 279 bytes | 279.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
remote: error: could not sync loose ref 'refs/heads/client_branch':
Bad file descriptor
when running "make test" (the above is from t5702 but I wouldn't be
surprised if this broke ALL ref updates).
Just before write_ref_to_lockfile() calls close_ref_gently() would
be a good place to make the fsync_loose_ref() call, perhaps?
Thanks.
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH 8/8] core.fsync: new option to harden packed references
2022-03-10 18:28 ` Junio C Hamano
@ 2022-03-11 9:10 ` Patrick Steinhardt
0 siblings, 0 replies; 122+ messages in thread
From: Patrick Steinhardt @ 2022-03-11 9:10 UTC (permalink / raw)
To: Junio C Hamano
Cc: git, rsbecker, bagasdotme, newren, avarab, nksingh85, sandals,
Neeraj K. Singh
[-- Attachment #1: Type: text/plain, Size: 1313 bytes --]
On Thu, Mar 10, 2022 at 10:28:57AM -0800, Junio C Hamano wrote:
> Patrick Steinhardt <ps@pks.im> writes:
>
> > diff --git a/refs/packed-backend.c b/refs/packed-backend.c
> > index 27dd8c3922..32d6635969 100644
> > --- a/refs/packed-backend.c
> > +++ b/refs/packed-backend.c
> > @@ -1262,7 +1262,8 @@ static int write_with_updates(struct packed_ref_store *refs,
> > goto error;
> > }
> >
> > - if (close_tempfile_gently(refs->tempfile)) {
> > + if (fsync_component(FSYNC_COMPONENT_PACKED_REFS, get_tempfile_fd(refs->tempfile)) ||
> > + close_tempfile_gently(refs->tempfile)) {
> > strbuf_addf(err, "error closing file %s: %s",
> > get_tempfile_path(refs->tempfile),
> > strerror(errno));
>
> I do not necessarily agree with the organization to have it as a
> component that is separate from other ref backends, but it is
> very pleasing to see that there is only one fsync necessary for the
> packed backend.
>
> Nice.
I was mostly adapting to the precedent set by Neeraj, where we also
distinguish loose objects and packed objects. Personally I don't mind
much whether we want to discern those two cases, and I'd be happy to
just merge them into a single "refs" knob. We can still split these up
at a later point if the need ever arises.
Patrick
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH 7/8] core.fsync: new option to harden loose references
2022-03-11 6:40 ` Junio C Hamano
@ 2022-03-11 9:15 ` Patrick Steinhardt
2022-03-11 9:36 ` Ævar Arnfjörð Bjarmason
0 siblings, 1 reply; 122+ messages in thread
From: Patrick Steinhardt @ 2022-03-11 9:15 UTC (permalink / raw)
To: Junio C Hamano
Cc: git, rsbecker, bagasdotme, newren, avarab, nksingh85, sandals,
Neeraj K. Singh
[-- Attachment #1: Type: text/plain, Size: 2202 bytes --]
On Thu, Mar 10, 2022 at 10:40:07PM -0800, Junio C Hamano wrote:
> Patrick Steinhardt <ps@pks.im> writes:
>
> > @@ -1504,6 +1513,7 @@ static int files_copy_or_rename_ref(struct ref_store *ref_store,
> > oidcpy(&lock->old_oid, &orig_oid);
> >
> > if (write_ref_to_lockfile(lock, &orig_oid, 0, &err) ||
> > + files_sync_loose_ref(lock, &err) ||
> > commit_ref_update(refs, lock, &orig_oid, logmsg, &err)) {
> > error("unable to write current sha1 into %s: %s", newrefname, err.buf);
> > strbuf_release(&err);
>
> Given that write_ref_to_lockfile() on the success code path does this:
>
> fd = get_lock_file_fd(&lock->lk);
> if (write_in_full(fd, oid_to_hex(oid), the_hash_algo->hexsz) < 0 ||
> write_in_full(fd, &term, 1) < 0 ||
> close_ref_gently(lock) < 0) {
> strbuf_addf(err,
> "couldn't write '%s'", get_lock_file_path(&lock->lk));
> unlock_ref(lock);
> return -1;
> }
> return 0;
>
> the above unfortunately does not work. By the time the new call to
> files_sync_loose_ref() is made, lock->fd is closed by the call to
> close_lock_file_gently() made in close_ref_gently(), and because of
> that, you'll get an error like this:
>
> Writing objects: 100% (3/3), 279 bytes | 279.00 KiB/s, done.
> Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
> remote: error: could not sync loose ref 'refs/heads/client_branch':
> Bad file descriptor
>
> when running "make test" (the above is from t5702 but I wouldn't be
> surprised if this broke ALL ref updates).
>
> Just before write_ref_to_lockfile() calls close_ref_gently() would
> be a good place to make the fsync_loose_ref() call, perhaps?
>
>
> Thanks.
Yeah, that thought indeed occurred to me this night, too. I was hoping
that I could fix this before anybody noticed ;)
It's a bit unfortunate that we can't just defer this to a later place to
hopefully implement this more efficiently, but so be it. The alternative
would be to re-open all locked loose refs and then sync them to disk,
but this would likely be a lot more painful than just syncing them to
disk before closing it.
Will fix, thanks.
Patrick
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH 7/8] core.fsync: new option to harden loose references
2022-03-11 9:15 ` Patrick Steinhardt
@ 2022-03-11 9:36 ` Ævar Arnfjörð Bjarmason
0 siblings, 0 replies; 122+ messages in thread
From: Ævar Arnfjörð Bjarmason @ 2022-03-11 9:36 UTC (permalink / raw)
To: Patrick Steinhardt
Cc: Junio C Hamano, git, rsbecker, bagasdotme, newren, nksingh85,
sandals, Neeraj K. Singh
On Fri, Mar 11 2022, Patrick Steinhardt wrote:
> [[PGP Signed Part:Undecided]]
> On Thu, Mar 10, 2022 at 10:40:07PM -0800, Junio C Hamano wrote:
>> Patrick Steinhardt <ps@pks.im> writes:
>>
>> > @@ -1504,6 +1513,7 @@ static int files_copy_or_rename_ref(struct ref_store *ref_store,
>> > oidcpy(&lock->old_oid, &orig_oid);
>> >
>> > if (write_ref_to_lockfile(lock, &orig_oid, 0, &err) ||
>> > + files_sync_loose_ref(lock, &err) ||
>> > commit_ref_update(refs, lock, &orig_oid, logmsg, &err)) {
>> > error("unable to write current sha1 into %s: %s", newrefname, err.buf);
>> > strbuf_release(&err);
>>
>> Given that write_ref_to_lockfile() on the success code path does this:
>>
>> fd = get_lock_file_fd(&lock->lk);
>> if (write_in_full(fd, oid_to_hex(oid), the_hash_algo->hexsz) < 0 ||
>> write_in_full(fd, &term, 1) < 0 ||
>> close_ref_gently(lock) < 0) {
>> strbuf_addf(err,
>> "couldn't write '%s'", get_lock_file_path(&lock->lk));
>> unlock_ref(lock);
>> return -1;
>> }
>> return 0;
>>
>> the above unfortunately does not work. By the time the new call to
>> files_sync_loose_ref() is made, lock->fd is closed by the call to
>> close_lock_file_gently() made in close_ref_gently(), and because of
>> that, you'll get an error like this:
>>
>> Writing objects: 100% (3/3), 279 bytes | 279.00 KiB/s, done.
>> Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
>> remote: error: could not sync loose ref 'refs/heads/client_branch':
>> Bad file descriptor
>>
>> when running "make test" (the above is from t5702 but I wouldn't be
>> surprised if this broke ALL ref updates).
>>
>> Just before write_ref_to_lockfile() calls close_ref_gently() would
>> be a good place to make the fsync_loose_ref() call, perhaps?
>>
>>
>> Thanks.
>
> Yeah, that thought indeed occurred to me this night, too. I was hoping
> that I could fix this before anybody noticed ;)
>
> It's a bit unfortunate that we can't just defer this to a later place to
> hopefully implement this more efficiently, but so be it. The alternative
> would be to re-open all locked loose refs and then sync them to disk,
> but this would likely be a lot more painful than just syncing them to
> disk before closing it.
Aside: is open/write/close followed by open/fsync/close on the same file
portably guaranteed to yield the same end result as a single
open/write/fsync/close?
I think in practice nobody would be insane enough to implement a system
to do otherwise, but on the other hand I've seen some really insane
behavior :)
I could see it being different e.g. in some NFS cases/configurations
where the fsync() for an open FD syncs to the remote storage, and the
second open() might therefore get the old version and noop-sync that.
Most implementations would guard against that in the common case by
having a local cache of outstanding data to flush, but if you're talking
to some sharded storage array for each request...
Anyway, I *think* it should be OK, just an aside to check the assumption
for any future work... :)
^ permalink raw reply [flat|nested] 122+ messages in thread
* [PATCH v2] core.fsync: new option to harden references
2022-03-10 22:43 ` [PATCH v6 0/6] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget
` (6 preceding siblings ...)
2022-03-10 23:34 ` [PATCH v6 0/6] A design for future-proofing fsync() configuration Junio C Hamano
@ 2022-03-11 9:58 ` Patrick Steinhardt
2022-03-25 6:11 ` SZEDER Gábor
7 siblings, 1 reply; 122+ messages in thread
From: Patrick Steinhardt @ 2022-03-11 9:58 UTC (permalink / raw)
To: git
Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, sandals,
Neeraj K. Singh, Junio C Hamano
[-- Attachment #1: Type: text/plain, Size: 5963 bytes --]
When writing both loose and packed references to disk we first create a
lockfile, write the updated values into that lockfile, and on commit we
rename the file into place. According to filesystem developers, this
behaviour is broken because applications should always sync data to disk
before doing the final rename to ensure data consistency [1][2][3]. If
applications fail to do this correctly, a hard crash of the machine can
easily result in corrupted on-disk data.
This kind of corruption can in fact be easily observed with Git when the
machine hard-resets shortly after writing references to disk. On
machines with ext4, this will likely lead to the "empty files" problem:
the file has been renamed, but its data has not been synced to disk. The
result is that the reference is corrupt, and in the worst case this can
lead to data loss.
Implement a new option to harden references so that users and admins can
avoid this scenario by syncing locked loose and packed references to
disk before we rename them into place.
[1]: https://thunk.org/tytso/blog/2009/03/15/dont-fear-the-fsync/
[2]: https://btrfs.wiki.kernel.org/index.php/FAQ (What are the crash guarantees of overwrite-by-rename)
[3]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/admin-guide/ext4.rst (see auto_da_alloc)
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
Hi,
here's my updated patch series which implements syncing of refs. It
applies on top of Neeraj's v6 of "A design for future-proofing fsync()
configuration".
I've simplified these patches a bit:
- I don't distinguishing between "loose" and "packed" refs anymore.
I agree with Junio that it's probably not worth it, but we can
still reintroduce the split at a later point without breaking
backwards compatibility if the need comes up.
- I've simplified the way loose refs are written to disk so that we
now sync them when before we close their files. The previous
implementation I had was broken because we tried to sync after
closing.
Because this really only changes a few lines of code I've also decided
to squash together the patches into a single one.
Patrick
Documentation/config/core.txt | 1 +
cache.h | 7 +++++--
config.c | 1 +
refs/files-backend.c | 1 +
refs/packed-backend.c | 3 ++-
5 files changed, 10 insertions(+), 3 deletions(-)
diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
index 37105a7be4..812cca7de7 100644
--- a/Documentation/config/core.txt
+++ b/Documentation/config/core.txt
@@ -575,6 +575,7 @@ but risks losing recent work in the event of an unclean system shutdown.
* `index` hardens the index when it is modified.
* `objects` is an aggregate option that is equivalent to
`loose-object,pack`.
+* `reference` hardens references modified in the repo.
* `derived-metadata` is an aggregate option that is equivalent to
`pack-metadata,commit-graph`.
* `committed` is an aggregate option that is currently equivalent to
diff --git a/cache.h b/cache.h
index cde0900d05..033e5b0779 100644
--- a/cache.h
+++ b/cache.h
@@ -1005,6 +1005,7 @@ enum fsync_component {
FSYNC_COMPONENT_PACK_METADATA = 1 << 2,
FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3,
FSYNC_COMPONENT_INDEX = 1 << 4,
+ FSYNC_COMPONENT_REFERENCE = 1 << 5,
};
#define FSYNC_COMPONENTS_OBJECTS (FSYNC_COMPONENT_LOOSE_OBJECT | \
@@ -1017,7 +1018,8 @@ enum fsync_component {
FSYNC_COMPONENTS_DERIVED_METADATA | \
~FSYNC_COMPONENT_LOOSE_OBJECT)
-#define FSYNC_COMPONENTS_COMMITTED (FSYNC_COMPONENTS_OBJECTS)
+#define FSYNC_COMPONENTS_COMMITTED (FSYNC_COMPONENTS_OBJECTS | \
+ FSYNC_COMPONENT_REFERENCE)
#define FSYNC_COMPONENTS_ADDED (FSYNC_COMPONENTS_COMMITTED | \
FSYNC_COMPONENT_INDEX)
@@ -1026,7 +1028,8 @@ enum fsync_component {
FSYNC_COMPONENT_PACK | \
FSYNC_COMPONENT_PACK_METADATA | \
FSYNC_COMPONENT_COMMIT_GRAPH | \
- FSYNC_COMPONENT_INDEX)
+ FSYNC_COMPONENT_INDEX | \
+ FSYNC_COMPONENT_REFERENCE)
/*
* A bitmask indicating which components of the repo should be fsynced.
diff --git a/config.c b/config.c
index eb75f65338..3c9b6b589a 100644
--- a/config.c
+++ b/config.c
@@ -1333,6 +1333,7 @@ static const struct fsync_component_name {
{ "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH },
{ "index", FSYNC_COMPONENT_INDEX },
{ "objects", FSYNC_COMPONENTS_OBJECTS },
+ { "reference", FSYNC_COMPONENT_REFERENCE },
{ "derived-metadata", FSYNC_COMPONENTS_DERIVED_METADATA },
{ "committed", FSYNC_COMPONENTS_COMMITTED },
{ "added", FSYNC_COMPONENTS_ADDED },
diff --git a/refs/files-backend.c b/refs/files-backend.c
index f59589d6cc..6521ee8af5 100644
--- a/refs/files-backend.c
+++ b/refs/files-backend.c
@@ -1787,6 +1787,7 @@ static int write_ref_to_lockfile(struct ref_lock *lock,
fd = get_lock_file_fd(&lock->lk);
if (write_in_full(fd, oid_to_hex(oid), the_hash_algo->hexsz) < 0 ||
write_in_full(fd, &term, 1) < 0 ||
+ fsync_component(FSYNC_COMPONENT_REFERENCE, get_lock_file_fd(&lock->lk)) < 0 ||
close_ref_gently(lock) < 0) {
strbuf_addf(err,
"couldn't write '%s'", get_lock_file_path(&lock->lk));
diff --git a/refs/packed-backend.c b/refs/packed-backend.c
index 27dd8c3922..9d704ccd3e 100644
--- a/refs/packed-backend.c
+++ b/refs/packed-backend.c
@@ -1262,7 +1262,8 @@ static int write_with_updates(struct packed_ref_store *refs,
goto error;
}
- if (close_tempfile_gently(refs->tempfile)) {
+ if (fsync_component(FSYNC_COMPONENT_REFERENCE, get_tempfile_fd(refs->tempfile)) ||
+ close_tempfile_gently(refs->tempfile)) {
strbuf_addf(err, "error closing file %s: %s",
get_tempfile_path(refs->tempfile),
strerror(errno));
--
2.35.1
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply related [flat|nested] 122+ messages in thread
* Re: [PATCH v6 0/6] A design for future-proofing fsync() configuration
2022-03-11 0:03 ` Neeraj Singh
@ 2022-03-11 18:50 ` Neeraj Singh
0 siblings, 0 replies; 122+ messages in thread
From: Neeraj Singh @ 2022-03-11 18:50 UTC (permalink / raw)
To: Junio C Hamano
Cc: Neeraj K. Singh via GitGitGadget, Git List, Randall S. Becker,
Bagas Sanjaya, Elijah Newren,
Ævar Arnfjörð Bjarmason, Patrick Steinhardt,
brian m. carlson, Johannes Schindelin, Neeraj K. Singh
On Thu, Mar 10, 2022 at 4:03 PM Neeraj Singh <nksingh85@gmail.com> wrote:
>
> On Thu, Mar 10, 2022 at 3:35 PM Junio C Hamano <gitster@pobox.com> wrote:
> >
> > "Neeraj K. Singh via GitGitGadget" <gitgitgadget@gmail.com> writes:
> >
> > > After this change, new persistent data files added to the repo will need to
> > > be added to the fsync_component enum and documented in the
> > > Documentation/config/core.txt text.
> > >
> > > V6 changes:
> > >
> > > * Move only the windows csprng includes into wrapper.c rather than all of
> > > them. This fixes the specific build issue due to broken Windows headers.
> > > [6]
> > > * Split the configuration parsing of core.fsync from the mechanism to focus
> > > the review.
> > > * Incorporate Patrick's patch at [7] into the core.fsync mechanism patch.
> > > * Pick the stricter one of core.fsyncObjectFiles and (fsync_components &
> > > FSYNC_COMPONENT_LOOSE_OBJECTS), to respect the older setting.
> > > * Issue a deprecation warning but keep parsing and honoring
> > > core.fsyncObjectFiles.
> > > * Change configuration parsing of core.fsync to always start with the
> > > platform default. none resets to the empty set. The comma separated list
> > > implies a set without regards to ordering now. This follows Junio's
> > > suggestion in [8].
> > > * Change the documentation of the core.fsync option to reflect the way the
> > > new parsing code works.
> >
> > Hmph, this seems to make one test fail.
> >
> > t5801-remote-helpers.sh (Wstat: 256 Tests: 31 Failed: 4)
> > Failed tests: 14-16, 31
> > Non-zero exit status: 1
> > Files=1, Tests=31, 2 wallclock secs ( 0.04 usr 0.00 sys + 1.40 cusr 1.62 csys = 3.06 CPU)
> > Result: FAIL
>
> Thanks for reporting this. I didn't see a failure in CI, nor when
> running that specific test in mingw. I also munged my config to
> include core.fsyncObjectFiles and didn't see a failure.
>
> Could you please share some more verbose output of the test, so I can
> look a bit deeper? In parallel, I'm trying again after merging my
> changes onto seen.
>
> Thanks,
> Neeraj
Hi Junio,
I've also tested v6-on-seen on Linux and I'm still not seeing the
failure. Does the
failure still happen on your end?
Thanks,
Neeraj
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v6 0/6] A design for future-proofing fsync() configuration
2022-03-10 23:34 ` [PATCH v6 0/6] A design for future-proofing fsync() configuration Junio C Hamano
2022-03-11 0:03 ` Neeraj Singh
@ 2022-03-13 23:50 ` Junio C Hamano
1 sibling, 0 replies; 122+ messages in thread
From: Junio C Hamano @ 2022-03-13 23:50 UTC (permalink / raw)
To: Neeraj K. Singh via GitGitGadget
Cc: git, rsbecker, bagasdotme, newren, avarab, nksingh85, ps, sandals,
Johannes Schindelin, Neeraj K. Singh
Junio C Hamano <gitster@pobox.com> writes:
> Hmph, this seems to make one test fail.
>
> t5801-remote-helpers.sh (Wstat: 256 Tests: 31 Failed: 4)
> Failed tests: 14-16, 31
> Non-zero exit status: 1
> Files=1, Tests=31, 2 wallclock secs ( 0.04 usr 0.00 sys + 1.40 cusr 1.62 csys = 3.06 CPU)
> Result: FAIL
False alarm. This byitself, or merged to 'seen' with other random
topics, no longer seem to break these tests.
^ permalink raw reply [flat|nested] 122+ messages in thread
* [PATCH v7] core.fsync: documentation and user-friendly aggregate options
2022-03-10 22:43 ` [PATCH v6 6/6] core.fsync: documentation and user-friendly aggregate options Neeraj Singh via GitGitGadget
@ 2022-03-15 19:12 ` Neeraj Singh
2022-03-15 19:32 ` Junio C Hamano
2022-03-23 14:20 ` do we have too much fsync() configuration in 'next'? (was: [PATCH v7] core.fsync: documentation and user-friendly aggregate options) Ævar Arnfjörð Bjarmason
0 siblings, 2 replies; 122+ messages in thread
From: Neeraj Singh @ 2022-03-15 19:12 UTC (permalink / raw)
To: gitgitgadget
Cc: Johannes.Schindelin, avarab, bagasdotme, git, neerajsi, newren,
nksingh85, ps, rsbecker, sandals
This commit adds aggregate options for the core.fsync setting that are
more user-friendly. These options are specified in terms of 'levels of
safety', indicating which Git operations are considered to be sync
points for durability.
The new documentation is also included here in its entirety for ease of
review.
Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
---
This revision fixes a grammatical mistake in the core.fsync documentation.
Documentation/config/core.txt | 40 +++++++++++++++++++++++++++++++++++
cache.h | 23 +++++++++++++++++---
config.c | 5 +++++
3 files changed, 65 insertions(+), 3 deletions(-)
diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
index ab911d6e26..9a3ad71e9e 100644
--- a/Documentation/config/core.txt
+++ b/Documentation/config/core.txt
@@ -547,6 +547,46 @@ core.whitespace::
is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent`
errors. The default tab width is 8. Allowed values are 1 to 63.
+core.fsync::
+ A comma-separated list of components of the repository that
+ should be hardened via the core.fsyncMethod when created or
+ modified. You can disable hardening of any component by
+ prefixing it with a '-'. Items that are not hardened may be
+ lost in the event of an unclean system shutdown. Unless you
+ have special requirements, it is recommended that you leave
+ this option empty or pick one of `committed`, `added`,
+ or `all`.
++
+When this configuration is encountered, the set of components starts with
+the platform default value, disabled components are removed, and additional
+components are added. `none` resets the state so that the platform default
+is ignored.
++
+The empty string resets the fsync configuration to the platform
+default. The default on most platforms is equivalent to
+`core.fsync=committed,-loose-object`, which has good performance,
+but risks losing recent work in the event of an unclean system shutdown.
++
+* `none` clears the set of fsynced components.
+* `loose-object` hardens objects added to the repo in loose-object form.
+* `pack` hardens objects added to the repo in packfile form.
+* `pack-metadata` hardens packfile bitmaps and indexes.
+* `commit-graph` hardens the commit graph file.
+* `index` hardens the index when it is modified.
+* `objects` is an aggregate option that is equivalent to
+ `loose-object,pack`.
+* `derived-metadata` is an aggregate option that is equivalent to
+ `pack-metadata,commit-graph`.
+* `committed` is an aggregate option that is currently equivalent to
+ `objects`. This mode sacrifices some performance to ensure that work
+ that is committed to the repository with `git commit` or similar commands
+ is hardened.
+* `added` is an aggregate option that is currently equivalent to
+ `committed,index`. This mode sacrifices additional performance to
+ ensure that the results of commands like `git add` and similar operations
+ are hardened.
+* `all` is an aggregate option that syncs all individual components above.
+
core.fsyncMethod::
A value indicating the strategy Git will use to harden repository data
using fsync and related primitives.
diff --git a/cache.h b/cache.h
index e08eeac6c1..86680f144e 100644
--- a/cache.h
+++ b/cache.h
@@ -1007,9 +1007,26 @@ enum fsync_component {
FSYNC_COMPONENT_INDEX = 1 << 4,
};
-#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \
- FSYNC_COMPONENT_PACK_METADATA | \
- FSYNC_COMPONENT_COMMIT_GRAPH)
+#define FSYNC_COMPONENTS_OBJECTS (FSYNC_COMPONENT_LOOSE_OBJECT | \
+ FSYNC_COMPONENT_PACK)
+
+#define FSYNC_COMPONENTS_DERIVED_METADATA (FSYNC_COMPONENT_PACK_METADATA | \
+ FSYNC_COMPONENT_COMMIT_GRAPH)
+
+#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENTS_OBJECTS | \
+ FSYNC_COMPONENTS_DERIVED_METADATA | \
+ ~FSYNC_COMPONENT_LOOSE_OBJECT)
+
+#define FSYNC_COMPONENTS_COMMITTED (FSYNC_COMPONENTS_OBJECTS)
+
+#define FSYNC_COMPONENTS_ADDED (FSYNC_COMPONENTS_COMMITTED | \
+ FSYNC_COMPONENT_INDEX)
+
+#define FSYNC_COMPONENTS_ALL (FSYNC_COMPONENT_LOOSE_OBJECT | \
+ FSYNC_COMPONENT_PACK | \
+ FSYNC_COMPONENT_PACK_METADATA | \
+ FSYNC_COMPONENT_COMMIT_GRAPH | \
+ FSYNC_COMPONENT_INDEX)
/*
* A bitmask indicating which components of the repo should be fsynced.
diff --git a/config.c b/config.c
index 80f33c9198..fd8e165931 100644
--- a/config.c
+++ b/config.c
@@ -1332,6 +1332,11 @@ static const struct fsync_component_name {
{ "pack-metadata", FSYNC_COMPONENT_PACK_METADATA },
{ "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH },
{ "index", FSYNC_COMPONENT_INDEX },
+ { "objects", FSYNC_COMPONENTS_OBJECTS },
+ { "derived-metadata", FSYNC_COMPONENTS_DERIVED_METADATA },
+ { "committed", FSYNC_COMPONENTS_COMMITTED },
+ { "added", FSYNC_COMPONENTS_ADDED },
+ { "all", FSYNC_COMPONENTS_ALL },
};
static enum fsync_component parse_fsync_components(const char *var, const char *string)
--
2.33.0.334.g55a40fc8fd
^ permalink raw reply related [flat|nested] 122+ messages in thread
* Re: [PATCH v7] core.fsync: documentation and user-friendly aggregate options
2022-03-15 19:12 ` [PATCH v7] " Neeraj Singh
@ 2022-03-15 19:32 ` Junio C Hamano
2022-03-15 19:56 ` Neeraj Singh
2022-03-23 14:20 ` do we have too much fsync() configuration in 'next'? (was: [PATCH v7] core.fsync: documentation and user-friendly aggregate options) Ævar Arnfjörð Bjarmason
1 sibling, 1 reply; 122+ messages in thread
From: Junio C Hamano @ 2022-03-15 19:32 UTC (permalink / raw)
To: Neeraj Singh
Cc: gitgitgadget, Johannes.Schindelin, avarab, bagasdotme, git,
neerajsi, newren, ps, rsbecker, sandals
Neeraj Singh <nksingh85@gmail.com> writes:
> This commit adds aggregate options for the core.fsync setting that are
> more user-friendly. These options are specified in terms of 'levels of
> safety', indicating which Git operations are considered to be sync
> points for durability.
>
> The new documentation is also included here in its entirety for ease of
> review.
>
> Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
> ---
> This revision fixes a grammatical mistake in the core.fsync documentation.
Is this meant to be [PATCH v7 6/6] where 1-5/6 of v7 are supposed to
be identical to their counterparts of v6 and therefor not sent? Not
complaining, just double-checking, as I do not want to assume and
end up missing the final updates to the other 5.
In the meantime, I'd assume that it is the case, and will fix the
author ident (you sent this from your gmail address) before
replacing the last bit.
The only change from the previous round is "the platform default on
most platform" -> "the default on most platforms", which looks
sensible.
Thanks.
1: 39f4b94c2c ! 1: dfeab99d23 core.fsync: documentation and user-friendly aggregate options
@@
## Metadata ##
-Author: Neeraj Singh <neerajsi@microsoft.com>
+Author: Neeraj Singh <nksingh85@gmail.com>
## Commit message ##
core.fsync: documentation and user-friendly aggregate options
@@ Documentation/config/core.txt: core.whitespace::
+is ignored.
++
+The empty string resets the fsync configuration to the platform
-+default. The platform default on most platform is equivalent to
++default. The default on most platforms is equivalent to
+`core.fsync=committed,-loose-object`, which has good performance,
+but risks losing recent work in the event of an unclean system shutdown.
++
Thanks.
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v7] core.fsync: documentation and user-friendly aggregate options
2022-03-15 19:32 ` Junio C Hamano
@ 2022-03-15 19:56 ` Neeraj Singh
0 siblings, 0 replies; 122+ messages in thread
From: Neeraj Singh @ 2022-03-15 19:56 UTC (permalink / raw)
To: Junio C Hamano
Cc: Neeraj K. Singh via GitGitGadget, Johannes Schindelin,
Ævar Arnfjörð Bjarmason, Bagas Sanjaya, Git List,
Neeraj Singh, Elijah Newren, Patrick Steinhardt,
Randall S. Becker, brian m. carlson
On Tue, Mar 15, 2022 at 12:32 PM Junio C Hamano <gitster@pobox.com> wrote:
>
> Neeraj Singh <nksingh85@gmail.com> writes:
>
> > This commit adds aggregate options for the core.fsync setting that are
> > more user-friendly. These options are specified in terms of 'levels of
> > safety', indicating which Git operations are considered to be sync
> > points for durability.
> >
> > The new documentation is also included here in its entirety for ease of
> > review.
> >
> > Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>
> > ---
> > This revision fixes a grammatical mistake in the core.fsync documentation.
>
> Is this meant to be [PATCH v7 6/6] where 1-5/6 of v7 are supposed to
> be identical to their counterparts of v6 and therefor not sent? Not
> complaining, just double-checking, as I do not want to assume and
> end up missing the final updates to the other 5.
>
> In the meantime, I'd assume that it is the case, and will fix the
> author ident (you sent this from your gmail address) before
> replacing the last bit.
>
> The only change from the previous round is "the platform default on
> most platform" -> "the default on most platforms", which looks
> sensible.
>
> Thanks.
Yes, that's right. I was trying out git-send-email for the first
time. I didn't want to spam the list with a whole new series where
only one patch had a change.
>
> 1: 39f4b94c2c ! 1: dfeab99d23 core.fsync: documentation and user-friendly aggregate options
> @@
> ## Metadata ##
> -Author: Neeraj Singh <neerajsi@microsoft.com>
> +Author: Neeraj Singh <nksingh85@gmail.com>
>
> ## Commit message ##
> core.fsync: documentation and user-friendly aggregate options
> @@ Documentation/config/core.txt: core.whitespace::
> +is ignored.
> ++
> +The empty string resets the fsync configuration to the platform
> -+default. The platform default on most platform is equivalent to
> ++default. The default on most platforms is equivalent to
> +`core.fsync=committed,-loose-object`, which has good performance,
> +but risks losing recent work in the event of an unclean system shutdown.
> ++
>
> Thanks.
^ permalink raw reply [flat|nested] 122+ messages in thread
* do we have too much fsync() configuration in 'next'? (was: [PATCH v7] core.fsync: documentation and user-friendly aggregate options)
2022-03-15 19:12 ` [PATCH v7] " Neeraj Singh
2022-03-15 19:32 ` Junio C Hamano
@ 2022-03-23 14:20 ` Ævar Arnfjörð Bjarmason
2022-03-25 21:24 ` Neeraj Singh
1 sibling, 1 reply; 122+ messages in thread
From: Ævar Arnfjörð Bjarmason @ 2022-03-23 14:20 UTC (permalink / raw)
To: Neeraj Singh
Cc: gitgitgadget, Johannes.Schindelin, bagasdotme, git, neerajsi,
newren, ps, rsbecker, sandals
On Tue, Mar 15 2022, Neeraj Singh wrote:
I know this is probably 80% my fault by egging you on about initially
adding the wildmatch() based thing you didn't go for.
But having looked at this with fresh eyes quite deeply I really think
we're severely over-configuring things here:
> +core.fsync::
> + A comma-separated list of components of the repository that
> + should be hardened via the core.fsyncMethod when created or
> + modified. You can disable hardening of any component by
> + prefixing it with a '-'. Items that are not hardened may be
> + lost in the event of an unclean system shutdown. Unless you
> + have special requirements, it is recommended that you leave
> + this option empty or pick one of `committed`, `added`,
> + or `all`.
> ++
> +When this configuration is encountered, the set of components starts with
> +the platform default value, disabled components are removed, and additional
> +components are added. `none` resets the state so that the platform default
> +is ignored.
> ++
> +The empty string resets the fsync configuration to the platform
> +default. The default on most platforms is equivalent to
> +`core.fsync=committed,-loose-object`, which has good performance,
> +but risks losing recent work in the event of an unclean system shutdown.
> ++
> +* `none` clears the set of fsynced components.
> +* `loose-object` hardens objects added to the repo in loose-object form.
> +* `pack` hardens objects added to the repo in packfile form.
> +* `pack-metadata` hardens packfile bitmaps and indexes.
> +* `commit-graph` hardens the commit graph file.
> +* `index` hardens the index when it is modified.
> +* `objects` is an aggregate option that is equivalent to
> + `loose-object,pack`.
> +* `derived-metadata` is an aggregate option that is equivalent to
> + `pack-metadata,commit-graph`.
> +* `committed` is an aggregate option that is currently equivalent to
> + `objects`. This mode sacrifices some performance to ensure that work
> + that is committed to the repository with `git commit` or similar commands
> + is hardened.
> +* `added` is an aggregate option that is currently equivalent to
> + `committed,index`. This mode sacrifices additional performance to
> + ensure that the results of commands like `git add` and similar operations
> + are hardened.
> +* `all` is an aggregate option that syncs all individual components above.
> +
> core.fsyncMethod::
> A value indicating the strategy Git will use to harden repository data
> using fsync and related primitives.
On top of my
https://lore.kernel.org/git/RFC-patch-v2-7.7-a5951366c6e-20220323T140753Z-avarab@gmail.com/
which makes the tmp-objdir part of your not-in-next-just-seen follow-up
series configurable via "fsyncMethod.batch.quarantine" I really think we
should just go for something like the belwo patch (note that
misspelled/mistook "bulk" for "batch" in that linked-t patch, fixed
below.
I.e. I think we should just do our default fsync() of everything, and
probably SOON make the fsync-ing of loose objects the default. Those who
care about performance will have "batch" (or "writeout-only"), which we
can have OS-specific detections for.
But really, all of the rest of this is unduly boxing us into
overconfiguration that I think nobody really needs.
If someone really needs this level of detail they can LD_PRELOAD
something to have fsync intercept fd's and paths, and act appropriately.
Worse, as the RFC series I sent
(https://lore.kernel.org/git/RFC-cover-v2-0.7-00000000000-20220323T140753Z-avarab@gmail.com/)
shows we can and should "batch" up fsync() operations across these
configuration boundaries, which this level of configuration would seem
to preclude.
Or, we'd need to explain why "core.fsync=loose-object" won't *actually*
call fsync() on a single loose object's fd under "batch" as I had to do
on top of this in
https://lore.kernel.org/git/RFC-patch-v2-6.7-c20301d7967-20220323T140753Z-avarab@gmail.com/
The same is going to apply for almost all of the rest of these
configuration categories.
I.e. a natural follow-up to e.g. batching across objects & index as I'm
doing in
https://lore.kernel.org/git/RFC-patch-v2-4.7-61f4f3d7ef4-20220323T140753Z-avarab@gmail.com/
is to do likewise for all the PACK-related stuff before we rename it
in-place. Or even have "git gc" issue only a single fsync() for all of
PACKs, their metadata files, commit-graph etc., and then rename() things
in-place as appropriate afterwards.
diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
index 365a12dc7ae..536238e209b 100644
--- a/Documentation/config/core.txt
+++ b/Documentation/config/core.txt
@@ -548,49 +548,35 @@ core.whitespace::
errors. The default tab width is 8. Allowed values are 1 to 63.
core.fsync::
- A comma-separated list of components of the repository that
- should be hardened via the core.fsyncMethod when created or
- modified. You can disable hardening of any component by
- prefixing it with a '-'. Items that are not hardened may be
- lost in the event of an unclean system shutdown. Unless you
- have special requirements, it is recommended that you leave
- this option empty or pick one of `committed`, `added`,
- or `all`.
-+
-When this configuration is encountered, the set of components starts with
-the platform default value, disabled components are removed, and additional
-components are added. `none` resets the state so that the platform default
-is ignored.
-+
-The empty string resets the fsync configuration to the platform
-default. The default on most platforms is equivalent to
-`core.fsync=committed,-loose-object`, which has good performance,
-but risks losing recent work in the event of an unclean system shutdown.
-+
-* `none` clears the set of fsynced components.
-* `loose-object` hardens objects added to the repo in loose-object form.
-* `pack` hardens objects added to the repo in packfile form.
-* `pack-metadata` hardens packfile bitmaps and indexes.
-* `commit-graph` hardens the commit graph file.
-* `index` hardens the index when it is modified.
-* `objects` is an aggregate option that is equivalent to
- `loose-object,pack`.
-* `derived-metadata` is an aggregate option that is equivalent to
- `pack-metadata,commit-graph`.
-* `committed` is an aggregate option that is currently equivalent to
- `objects`. This mode sacrifices some performance to ensure that work
- that is committed to the repository with `git commit` or similar commands
- is hardened.
-* `added` is an aggregate option that is currently equivalent to
- `committed,index`. This mode sacrifices additional performance to
- ensure that the results of commands like `git add` and similar operations
- are hardened.
-* `all` is an aggregate option that syncs all individual components above.
+ A boolen defaulting to `true`. To ensure data integrity git
+ will fsync() its objects, index and refu updates etc. This can
+ be set to `false` to disable `fsync()`-ing.
++
+Only set this to `false` if you know what you're doing, and are
+prepared to deal with data corruption. Valid use-cases include
+throwaway uses of repositories on ramdisks, one-off mass-imports
+followed by calling `sync(1)` etc.
++
+Note that the syncing of loose objects is currently excluded from
+`core.fsync=true`. To turn on all fsync-ing you'll need
+`core.fsync=true` and `core.fsyncObjectFiles=true`, but see
+`core.fsyncMethod=batch` below for a much faster alternative that's
+just as safe on various modern OS's.
++
+The default is in flux and may change in the future, in particular the
+equivalent of the already-deprecated `core.fsyncObjectFiles` setting
+might soon default to `true`, and `core.fsyncMethod`'s default of
+`fsync` might default to a setting deemed to be safe on the local OS,
+suc has `batch` or `writeout-only`
core.fsyncMethod::
A value indicating the strategy Git will use to harden repository data
using fsync and related primitives.
+
+Defaults to `fsync`, but as discussed for `core.fsync` above might
+change to use one of the values below taking advantage of
+platform-specific "faster `fsync()`".
++
* `fsync` uses the fsync() system call or platform equivalents.
* `writeout-only` issues pagecache writeback requests, but depending on the
filesystem and storage hardware, data added to the repository may not be
@@ -680,8 +666,8 @@ backed up by any standard (e.g. POSIX), but worked in practice on some
Linux setups.
+
Nowadays you should almost certainly want to use
-`core.fsync=loose-object` instead in combination with
-`core.fsyncMethod=bulk`, and possibly with
+`core.fsync=true` instead in combination with
+`core.fsyncMethod=batch`, and possibly with
`fsyncMethod.batch.quarantine=true`, see above. On modern OS's (Linux,
OSX, Windows) that gives you most of the performance benefit of
`core.fsyncObjectFiles=false` with all of the safety of the old
^ permalink raw reply related [flat|nested] 122+ messages in thread
* Re: [PATCH v2] core.fsync: new option to harden references
2022-03-11 9:58 ` [PATCH v2] core.fsync: new option to harden references Patrick Steinhardt
@ 2022-03-25 6:11 ` SZEDER Gábor
0 siblings, 0 replies; 122+ messages in thread
From: SZEDER Gábor @ 2022-03-25 6:11 UTC (permalink / raw)
To: Patrick Steinhardt
Cc: git, rsbecker, bagasdotme, newren, avarab, nksingh85, sandals,
Neeraj K. Singh, Junio C Hamano
On Fri, Mar 11, 2022 at 10:58:59AM +0100, Patrick Steinhardt wrote:
> When writing both loose and packed references to disk we first create a
> lockfile, write the updated values into that lockfile, and on commit we
> rename the file into place. According to filesystem developers, this
> behaviour is broken because applications should always sync data to disk
> before doing the final rename to ensure data consistency [1][2][3]. If
> applications fail to do this correctly, a hard crash of the machine can
> easily result in corrupted on-disk data.
>
> This kind of corruption can in fact be easily observed with Git when the
> machine hard-resets shortly after writing references to disk. On
> machines with ext4, this will likely lead to the "empty files" problem:
> the file has been renamed, but its data has not been synced to disk. The
> result is that the reference is corrupt, and in the worst case this can
> lead to data loss.
>
> Implement a new option to harden references so that users and admins can
> avoid this scenario by syncing locked loose and packed references to
> disk before we rename them into place.
In 't5541-http-push-smart.sh' there is a test case called 'push 2000
tags over http', which does pretty much what it's title says. This
patch makes that test case significantly slower.
diff --git a/t/t5541-http-push-smart.sh b/t/t5541-http-push-smart.sh
index 8ca50f8b18..d7e94cb791 100755
--- a/t/t5541-http-push-smart.sh
+++ b/t/t5541-http-push-smart.sh
@@ -415,7 +415,7 @@ test_expect_success CMDLINE_LIMIT 'push 2000 tags over http' '
sort |
sed "s|.*|$sha1 refs/tags/really-long-tag-name-&|" \
>.git/packed-refs &&
- run_with_limited_cmdline git push --mirror
+ run_with_limited_cmdline /usr/bin/time git push --mirror
'
test_expect_success GPG 'push with post-receive to inspect certificate' '
Before this patch (bc22d845c4^) 'time' reports:
3.62user 0.03system 0:03.83elapsed 95%CPU (0avgtext+0avgdata 11904maxresident)k
0inputs+312outputs (0major+4597minor)pagefaults 0swaps
With this patch (bc22d845c4):
3.56user 0.04system 0:33.60elapsed 10%CPU (0avgtext+0avgdata 11832maxresident)k
0inputs+320outputs (0major+4578minor)pagefaults 0swaps
And the total runtime of the whole test script increases from 8-9s to
37-39s.
I wonder whether we should relax the fsync options for this test case.
>
> [1]: https://thunk.org/tytso/blog/2009/03/15/dont-fear-the-fsync/
> [2]: https://btrfs.wiki.kernel.org/index.php/FAQ (What are the crash guarantees of overwrite-by-rename)
> [3]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/admin-guide/ext4.rst (see auto_da_alloc)
>
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
>
> Hi,
>
> here's my updated patch series which implements syncing of refs. It
> applies on top of Neeraj's v6 of "A design for future-proofing fsync()
> configuration".
>
> I've simplified these patches a bit:
>
> - I don't distinguishing between "loose" and "packed" refs anymore.
> I agree with Junio that it's probably not worth it, but we can
> still reintroduce the split at a later point without breaking
> backwards compatibility if the need comes up.
>
> - I've simplified the way loose refs are written to disk so that we
> now sync them when before we close their files. The previous
> implementation I had was broken because we tried to sync after
> closing.
>
> Because this really only changes a few lines of code I've also decided
> to squash together the patches into a single one.
>
> Patrick
>
> Documentation/config/core.txt | 1 +
> cache.h | 7 +++++--
> config.c | 1 +
> refs/files-backend.c | 1 +
> refs/packed-backend.c | 3 ++-
> 5 files changed, 10 insertions(+), 3 deletions(-)
>
> diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
> index 37105a7be4..812cca7de7 100644
> --- a/Documentation/config/core.txt
> +++ b/Documentation/config/core.txt
> @@ -575,6 +575,7 @@ but risks losing recent work in the event of an unclean system shutdown.
> * `index` hardens the index when it is modified.
> * `objects` is an aggregate option that is equivalent to
> `loose-object,pack`.
> +* `reference` hardens references modified in the repo.
> * `derived-metadata` is an aggregate option that is equivalent to
> `pack-metadata,commit-graph`.
> * `committed` is an aggregate option that is currently equivalent to
> diff --git a/cache.h b/cache.h
> index cde0900d05..033e5b0779 100644
> --- a/cache.h
> +++ b/cache.h
> @@ -1005,6 +1005,7 @@ enum fsync_component {
> FSYNC_COMPONENT_PACK_METADATA = 1 << 2,
> FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3,
> FSYNC_COMPONENT_INDEX = 1 << 4,
> + FSYNC_COMPONENT_REFERENCE = 1 << 5,
> };
>
> #define FSYNC_COMPONENTS_OBJECTS (FSYNC_COMPONENT_LOOSE_OBJECT | \
> @@ -1017,7 +1018,8 @@ enum fsync_component {
> FSYNC_COMPONENTS_DERIVED_METADATA | \
> ~FSYNC_COMPONENT_LOOSE_OBJECT)
>
> -#define FSYNC_COMPONENTS_COMMITTED (FSYNC_COMPONENTS_OBJECTS)
> +#define FSYNC_COMPONENTS_COMMITTED (FSYNC_COMPONENTS_OBJECTS | \
> + FSYNC_COMPONENT_REFERENCE)
>
> #define FSYNC_COMPONENTS_ADDED (FSYNC_COMPONENTS_COMMITTED | \
> FSYNC_COMPONENT_INDEX)
> @@ -1026,7 +1028,8 @@ enum fsync_component {
> FSYNC_COMPONENT_PACK | \
> FSYNC_COMPONENT_PACK_METADATA | \
> FSYNC_COMPONENT_COMMIT_GRAPH | \
> - FSYNC_COMPONENT_INDEX)
> + FSYNC_COMPONENT_INDEX | \
> + FSYNC_COMPONENT_REFERENCE)
>
> /*
> * A bitmask indicating which components of the repo should be fsynced.
> diff --git a/config.c b/config.c
> index eb75f65338..3c9b6b589a 100644
> --- a/config.c
> +++ b/config.c
> @@ -1333,6 +1333,7 @@ static const struct fsync_component_name {
> { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH },
> { "index", FSYNC_COMPONENT_INDEX },
> { "objects", FSYNC_COMPONENTS_OBJECTS },
> + { "reference", FSYNC_COMPONENT_REFERENCE },
> { "derived-metadata", FSYNC_COMPONENTS_DERIVED_METADATA },
> { "committed", FSYNC_COMPONENTS_COMMITTED },
> { "added", FSYNC_COMPONENTS_ADDED },
> diff --git a/refs/files-backend.c b/refs/files-backend.c
> index f59589d6cc..6521ee8af5 100644
> --- a/refs/files-backend.c
> +++ b/refs/files-backend.c
> @@ -1787,6 +1787,7 @@ static int write_ref_to_lockfile(struct ref_lock *lock,
> fd = get_lock_file_fd(&lock->lk);
> if (write_in_full(fd, oid_to_hex(oid), the_hash_algo->hexsz) < 0 ||
> write_in_full(fd, &term, 1) < 0 ||
> + fsync_component(FSYNC_COMPONENT_REFERENCE, get_lock_file_fd(&lock->lk)) < 0 ||
> close_ref_gently(lock) < 0) {
> strbuf_addf(err,
> "couldn't write '%s'", get_lock_file_path(&lock->lk));
> diff --git a/refs/packed-backend.c b/refs/packed-backend.c
> index 27dd8c3922..9d704ccd3e 100644
> --- a/refs/packed-backend.c
> +++ b/refs/packed-backend.c
> @@ -1262,7 +1262,8 @@ static int write_with_updates(struct packed_ref_store *refs,
> goto error;
> }
>
> - if (close_tempfile_gently(refs->tempfile)) {
> + if (fsync_component(FSYNC_COMPONENT_REFERENCE, get_tempfile_fd(refs->tempfile)) ||
> + close_tempfile_gently(refs->tempfile)) {
> strbuf_addf(err, "error closing file %s: %s",
> get_tempfile_path(refs->tempfile),
> strerror(errno));
> --
> 2.35.1
>
^ permalink raw reply related [flat|nested] 122+ messages in thread
* Re: do we have too much fsync() configuration in 'next'? (was: [PATCH v7] core.fsync: documentation and user-friendly aggregate options)
2022-03-23 14:20 ` do we have too much fsync() configuration in 'next'? (was: [PATCH v7] core.fsync: documentation and user-friendly aggregate options) Ævar Arnfjörð Bjarmason
@ 2022-03-25 21:24 ` Neeraj Singh
2022-03-26 0:24 ` Ævar Arnfjörð Bjarmason
0 siblings, 1 reply; 122+ messages in thread
From: Neeraj Singh @ 2022-03-25 21:24 UTC (permalink / raw)
To: Ævar Arnfjörð Bjarmason
Cc: Neeraj K. Singh via GitGitGadget, Johannes Schindelin,
Bagas Sanjaya, Git List, Neeraj Singh, Elijah Newren,
Patrick Steinhardt, Randall S. Becker, brian m. carlson
On Wed, Mar 23, 2022 at 7:46 AM Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
>
>
> On Tue, Mar 15 2022, Neeraj Singh wrote:
>
> I know this is probably 80% my fault by egging you on about initially
> adding the wildmatch() based thing you didn't go for.
>
> But having looked at this with fresh eyes quite deeply I really think
> we're severely over-configuring things here:
>
> > +core.fsync::
> > + A comma-separated list of components of the repository that
> > + should be hardened via the core.fsyncMethod when created or
> > + modified. You can disable hardening of any component by
> > + prefixing it with a '-'. Items that are not hardened may be
> > + lost in the event of an unclean system shutdown. Unless you
> > + have special requirements, it is recommended that you leave
> > + this option empty or pick one of `committed`, `added`,
> > + or `all`.
> > ++
> > +When this configuration is encountered, the set of components starts with
> > +the platform default value, disabled components are removed, and additional
> > +components are added. `none` resets the state so that the platform default
> > +is ignored.
> > ++
> > +The empty string resets the fsync configuration to the platform
> > +default. The default on most platforms is equivalent to
> > +`core.fsync=committed,-loose-object`, which has good performance,
> > +but risks losing recent work in the event of an unclean system shutdown.
> > ++
> > +* `none` clears the set of fsynced components.
> > +* `loose-object` hardens objects added to the repo in loose-object form.
> > +* `pack` hardens objects added to the repo in packfile form.
> > +* `pack-metadata` hardens packfile bitmaps and indexes.
> > +* `commit-graph` hardens the commit graph file.
> > +* `index` hardens the index when it is modified.
> > +* `objects` is an aggregate option that is equivalent to
> > + `loose-object,pack`.
> > +* `derived-metadata` is an aggregate option that is equivalent to
> > + `pack-metadata,commit-graph`.
> > +* `committed` is an aggregate option that is currently equivalent to
> > + `objects`. This mode sacrifices some performance to ensure that work
> > + that is committed to the repository with `git commit` or similar commands
> > + is hardened.
> > +* `added` is an aggregate option that is currently equivalent to
> > + `committed,index`. This mode sacrifices additional performance to
> > + ensure that the results of commands like `git add` and similar operations
> > + are hardened.
> > +* `all` is an aggregate option that syncs all individual components above.
> > +
> > core.fsyncMethod::
> > A value indicating the strategy Git will use to harden repository data
> > using fsync and related primitives.
>
> On top of my
> https://lore.kernel.org/git/RFC-patch-v2-7.7-a5951366c6e-20220323T140753Z-avarab@gmail.com/
> which makes the tmp-objdir part of your not-in-next-just-seen follow-up
> series configurable via "fsyncMethod.batch.quarantine" I really think we
> should just go for something like the belwo patch (note that
> misspelled/mistook "bulk" for "batch" in that linked-t patch, fixed
> below.
>
> I.e. I think we should just do our default fsync() of everything, and
> probably SOON make the fsync-ing of loose objects the default. Those who
> care about performance will have "batch" (or "writeout-only"), which we
> can have OS-specific detections for.
>
> But really, all of the rest of this is unduly boxing us into
> overconfiguration that I think nobody really needs.
>
We've gone over this a few times already, but just wanted to state it
again. The really detailed settings are really there for Git hosters
like GitLab or GitHub. I'd be happy to remove the per-component
core.fsync values from the documentation and leave just the ones we
point the user to.
> If someone really needs this level of detail they can LD_PRELOAD
> something to have fsync intercept fd's and paths, and act appropriately.
>
> Worse, as the RFC series I sent
> (https://lore.kernel.org/git/RFC-cover-v2-0.7-00000000000-20220323T140753Z-avarab@gmail.com/)
> shows we can and should "batch" up fsync() operations across these
> configuration boundaries, which this level of configuration would seem
> to preclude.
>
> Or, we'd need to explain why "core.fsync=loose-object" won't *actually*
> call fsync() on a single loose object's fd under "batch" as I had to do
> on top of this in
> https://lore.kernel.org/git/RFC-patch-v2-6.7-c20301d7967-20220323T140753Z-avarab@gmail.com/
>
99.9% of users don't care and won't look. The ones who do look deeper
and understand the issues have source code and access to this ML
discussion to understand why this works this way.
> The same is going to apply for almost all of the rest of these
> configuration categories.
>
> I.e. a natural follow-up to e.g. batching across objects & index as I'm
> doing in
> https://lore.kernel.org/git/RFC-patch-v2-4.7-61f4f3d7ef4-20220323T140753Z-avarab@gmail.com/
> is to do likewise for all the PACK-related stuff before we rename it
> in-place. Or even have "git gc" issue only a single fsync() for all of
> PACKs, their metadata files, commit-graph etc., and then rename() things
> in-place as appropriate afterwards.
>
> diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
> index 365a12dc7ae..536238e209b 100644
> --- a/Documentation/config/core.txt
> +++ b/Documentation/config/core.txt
> @@ -548,49 +548,35 @@ core.whitespace::
> errors. The default tab width is 8. Allowed values are 1 to 63.
>
> core.fsync::
> - A comma-separated list of components of the repository that
> - should be hardened via the core.fsyncMethod when created or
> - modified. You can disable hardening of any component by
> - prefixing it with a '-'. Items that are not hardened may be
> - lost in the event of an unclean system shutdown. Unless you
> - have special requirements, it is recommended that you leave
> - this option empty or pick one of `committed`, `added`,
> - or `all`.
> -+
> -When this configuration is encountered, the set of components starts with
> -the platform default value, disabled components are removed, and additional
> -components are added. `none` resets the state so that the platform default
> -is ignored.
> -+
> -The empty string resets the fsync configuration to the platform
> -default. The default on most platforms is equivalent to
> -`core.fsync=committed,-loose-object`, which has good performance,
> -but risks losing recent work in the event of an unclean system shutdown.
> -+
> -* `none` clears the set of fsynced components.
> -* `loose-object` hardens objects added to the repo in loose-object form.
> -* `pack` hardens objects added to the repo in packfile form.
> -* `pack-metadata` hardens packfile bitmaps and indexes.
> -* `commit-graph` hardens the commit graph file.
> -* `index` hardens the index when it is modified.
> -* `objects` is an aggregate option that is equivalent to
> - `loose-object,pack`.
> -* `derived-metadata` is an aggregate option that is equivalent to
> - `pack-metadata,commit-graph`.
> -* `committed` is an aggregate option that is currently equivalent to
> - `objects`. This mode sacrifices some performance to ensure that work
> - that is committed to the repository with `git commit` or similar commands
> - is hardened.
> -* `added` is an aggregate option that is currently equivalent to
> - `committed,index`. This mode sacrifices additional performance to
> - ensure that the results of commands like `git add` and similar operations
> - are hardened.
> -* `all` is an aggregate option that syncs all individual components above.
> + A boolen defaulting to `true`. To ensure data integrity git
> + will fsync() its objects, index and refu updates etc. This can
> + be set to `false` to disable `fsync()`-ing.
> ++
> +Only set this to `false` if you know what you're doing, and are
> +prepared to deal with data corruption. Valid use-cases include
> +throwaway uses of repositories on ramdisks, one-off mass-imports
> +followed by calling `sync(1)` etc.
> ++
> +Note that the syncing of loose objects is currently excluded from
> +`core.fsync=true`. To turn on all fsync-ing you'll need
> +`core.fsync=true` and `core.fsyncObjectFiles=true`, but see
> +`core.fsyncMethod=batch` below for a much faster alternative that's
> +just as safe on various modern OS's.
> ++
> +The default is in flux and may change in the future, in particular the
> +equivalent of the already-deprecated `core.fsyncObjectFiles` setting
> +might soon default to `true`, and `core.fsyncMethod`'s default of
> +`fsync` might default to a setting deemed to be safe on the local OS,
> +suc has `batch` or `writeout-only`
>
> core.fsyncMethod::
> A value indicating the strategy Git will use to harden repository data
> using fsync and related primitives.
> +
> +Defaults to `fsync`, but as discussed for `core.fsync` above might
> +change to use one of the values below taking advantage of
> +platform-specific "faster `fsync()`".
> ++
> * `fsync` uses the fsync() system call or platform equivalents.
> * `writeout-only` issues pagecache writeback requests, but depending on the
> filesystem and storage hardware, data added to the repository may not be
> @@ -680,8 +666,8 @@ backed up by any standard (e.g. POSIX), but worked in practice on some
> Linux setups.
> +
> Nowadays you should almost certainly want to use
> -`core.fsync=loose-object` instead in combination with
> -`core.fsyncMethod=bulk`, and possibly with
> +`core.fsync=true` instead in combination with
> +`core.fsyncMethod=batch`, and possibly with
> `fsyncMethod.batch.quarantine=true`, see above. On modern OS's (Linux,
> OSX, Windows) that gives you most of the performance benefit of
> `core.fsyncObjectFiles=false` with all of the safety of the old
I'm at the point where I don't want to endlessly revisit this discussion.
-Neeraj
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: do we have too much fsync() configuration in 'next'? (was: [PATCH v7] core.fsync: documentation and user-friendly aggregate options)
2022-03-25 21:24 ` Neeraj Singh
@ 2022-03-26 0:24 ` Ævar Arnfjörð Bjarmason
2022-03-26 1:23 ` do we have too much fsync() configuration in 'next'? Junio C Hamano
2022-03-26 1:25 ` do we have too much fsync() configuration in 'next'? (was: [PATCH v7] core.fsync: documentation and user-friendly aggregate options) Neeraj Singh
0 siblings, 2 replies; 122+ messages in thread
From: Ævar Arnfjörð Bjarmason @ 2022-03-26 0:24 UTC (permalink / raw)
To: Neeraj Singh
Cc: Neeraj K. Singh via GitGitGadget, Johannes Schindelin,
Bagas Sanjaya, Git List, Neeraj Singh, Elijah Newren,
Patrick Steinhardt, Randall S. Becker, brian m. carlson
On Fri, Mar 25 2022, Neeraj Singh wrote:
> On Wed, Mar 23, 2022 at 7:46 AM Ævar Arnfjörð Bjarmason
> <avarab@gmail.com> wrote:
>>
>>
>> On Tue, Mar 15 2022, Neeraj Singh wrote:
>>
>> I know this is probably 80% my fault by egging you on about initially
>> adding the wildmatch() based thing you didn't go for.
>>
>> But having looked at this with fresh eyes quite deeply I really think
>> we're severely over-configuring things here:
>>
>> > +core.fsync::
>> > + A comma-separated list of components of the repository that
>> > + should be hardened via the core.fsyncMethod when created or
>> > + modified. You can disable hardening of any component by
>> > + prefixing it with a '-'. Items that are not hardened may be
>> > + lost in the event of an unclean system shutdown. Unless you
>> > + have special requirements, it is recommended that you leave
>> > + this option empty or pick one of `committed`, `added`,
>> > + or `all`.
>> > ++
>> > +When this configuration is encountered, the set of components starts with
>> > +the platform default value, disabled components are removed, and additional
>> > +components are added. `none` resets the state so that the platform default
>> > +is ignored.
>> > ++
>> > +The empty string resets the fsync configuration to the platform
>> > +default. The default on most platforms is equivalent to
>> > +`core.fsync=committed,-loose-object`, which has good performance,
>> > +but risks losing recent work in the event of an unclean system shutdown.
>> > ++
>> > +* `none` clears the set of fsynced components.
>> > +* `loose-object` hardens objects added to the repo in loose-object form.
>> > +* `pack` hardens objects added to the repo in packfile form.
>> > +* `pack-metadata` hardens packfile bitmaps and indexes.
>> > +* `commit-graph` hardens the commit graph file.
>> > +* `index` hardens the index when it is modified.
>> > +* `objects` is an aggregate option that is equivalent to
>> > + `loose-object,pack`.
>> > +* `derived-metadata` is an aggregate option that is equivalent to
>> > + `pack-metadata,commit-graph`.
>> > +* `committed` is an aggregate option that is currently equivalent to
>> > + `objects`. This mode sacrifices some performance to ensure that work
>> > + that is committed to the repository with `git commit` or similar commands
>> > + is hardened.
>> > +* `added` is an aggregate option that is currently equivalent to
>> > + `committed,index`. This mode sacrifices additional performance to
>> > + ensure that the results of commands like `git add` and similar operations
>> > + are hardened.
>> > +* `all` is an aggregate option that syncs all individual components above.
>> > +
>> > core.fsyncMethod::
>> > A value indicating the strategy Git will use to harden repository data
>> > using fsync and related primitives.
>>
>> On top of my
>> https://lore.kernel.org/git/RFC-patch-v2-7.7-a5951366c6e-20220323T140753Z-avarab@gmail.com/
>> which makes the tmp-objdir part of your not-in-next-just-seen follow-up
>> series configurable via "fsyncMethod.batch.quarantine" I really think we
>> should just go for something like the belwo patch (note that
>> misspelled/mistook "bulk" for "batch" in that linked-t patch, fixed
>> below.
>>
>> I.e. I think we should just do our default fsync() of everything, and
>> probably SOON make the fsync-ing of loose objects the default. Those who
>> care about performance will have "batch" (or "writeout-only"), which we
>> can have OS-specific detections for.
>>
>> But really, all of the rest of this is unduly boxing us into
>> overconfiguration that I think nobody really needs.
>>
>
> We've gone over this a few times already, but just wanted to state it
> again. The really detailed settings are really there for Git hosters
> like GitLab or GitHub. I'd be happy to remove the per-component
> core.fsync values from the documentation and leave just the ones we
> point the user to.
I'm prettty sure (but Patrick knows more) that GitLab's plan for this is
to keep it at whatever the safest setting is, presumably GitHub's as
well (but I don't know at all on that front).
>> If someone really needs this level of detail they can LD_PRELOAD
>> something to have fsync intercept fd's and paths, and act appropriately.
>>
>> Worse, as the RFC series I sent
>> (https://lore.kernel.org/git/RFC-cover-v2-0.7-00000000000-20220323T140753Z-avarab@gmail.com/)
>> shows we can and should "batch" up fsync() operations across these
>> configuration boundaries, which this level of configuration would seem
>> to preclude.
>>
>> Or, we'd need to explain why "core.fsync=loose-object" won't *actually*
>> call fsync() on a single loose object's fd under "batch" as I had to do
>> on top of this in
>> https://lore.kernel.org/git/RFC-patch-v2-6.7-c20301d7967-20220323T140753Z-avarab@gmail.com/
>>
>
> 99.9% of users don't care and won't look. The ones who do look deeper
> and understand the issues have source code and access to this ML
> discussion to understand why this works this way.
Exactly, so we can hopefully have a simpler interface.
>> The same is going to apply for almost all of the rest of these
>> configuration categories.
>>
>> I.e. a natural follow-up to e.g. batching across objects & index as I'm
>> doing in
>> https://lore.kernel.org/git/RFC-patch-v2-4.7-61f4f3d7ef4-20220323T140753Z-avarab@gmail.com/
>> is to do likewise for all the PACK-related stuff before we rename it
>> in-place. Or even have "git gc" issue only a single fsync() for all of
>> PACKs, their metadata files, commit-graph etc., and then rename() things
>> in-place as appropriate afterwards.
>>
>> diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
>> index 365a12dc7ae..536238e209b 100644
>> --- a/Documentation/config/core.txt
>> +++ b/Documentation/config/core.txt
>> @@ -548,49 +548,35 @@ core.whitespace::
>> errors. The default tab width is 8. Allowed values are 1 to 63.
>>
>> core.fsync::
>> - A comma-separated list of components of the repository that
>> - should be hardened via the core.fsyncMethod when created or
>> - modified. You can disable hardening of any component by
>> - prefixing it with a '-'. Items that are not hardened may be
>> - lost in the event of an unclean system shutdown. Unless you
>> - have special requirements, it is recommended that you leave
>> - this option empty or pick one of `committed`, `added`,
>> - or `all`.
>> -+
>> -When this configuration is encountered, the set of components starts with
>> -the platform default value, disabled components are removed, and additional
>> -components are added. `none` resets the state so that the platform default
>> -is ignored.
>> -+
>> -The empty string resets the fsync configuration to the platform
>> -default. The default on most platforms is equivalent to
>> -`core.fsync=committed,-loose-object`, which has good performance,
>> -but risks losing recent work in the event of an unclean system shutdown.
>> -+
>> -* `none` clears the set of fsynced components.
>> -* `loose-object` hardens objects added to the repo in loose-object form.
>> -* `pack` hardens objects added to the repo in packfile form.
>> -* `pack-metadata` hardens packfile bitmaps and indexes.
>> -* `commit-graph` hardens the commit graph file.
>> -* `index` hardens the index when it is modified.
>> -* `objects` is an aggregate option that is equivalent to
>> - `loose-object,pack`.
>> -* `derived-metadata` is an aggregate option that is equivalent to
>> - `pack-metadata,commit-graph`.
>> -* `committed` is an aggregate option that is currently equivalent to
>> - `objects`. This mode sacrifices some performance to ensure that work
>> - that is committed to the repository with `git commit` or similar commands
>> - is hardened.
>> -* `added` is an aggregate option that is currently equivalent to
>> - `committed,index`. This mode sacrifices additional performance to
>> - ensure that the results of commands like `git add` and similar operations
>> - are hardened.
>> -* `all` is an aggregate option that syncs all individual components above.
>> + A boolen defaulting to `true`. To ensure data integrity git
>> + will fsync() its objects, index and refu updates etc. This can
>> + be set to `false` to disable `fsync()`-ing.
>> ++
>> +Only set this to `false` if you know what you're doing, and are
>> +prepared to deal with data corruption. Valid use-cases include
>> +throwaway uses of repositories on ramdisks, one-off mass-imports
>> +followed by calling `sync(1)` etc.
>> ++
>> +Note that the syncing of loose objects is currently excluded from
>> +`core.fsync=true`. To turn on all fsync-ing you'll need
>> +`core.fsync=true` and `core.fsyncObjectFiles=true`, but see
>> +`core.fsyncMethod=batch` below for a much faster alternative that's
>> +just as safe on various modern OS's.
>> ++
>> +The default is in flux and may change in the future, in particular the
>> +equivalent of the already-deprecated `core.fsyncObjectFiles` setting
>> +might soon default to `true`, and `core.fsyncMethod`'s default of
>> +`fsync` might default to a setting deemed to be safe on the local OS,
>> +suc has `batch` or `writeout-only`
>>
>> core.fsyncMethod::
>> A value indicating the strategy Git will use to harden repository data
>> using fsync and related primitives.
>> +
>> +Defaults to `fsync`, but as discussed for `core.fsync` above might
>> +change to use one of the values below taking advantage of
>> +platform-specific "faster `fsync()`".
>> ++
>> * `fsync` uses the fsync() system call or platform equivalents.
>> * `writeout-only` issues pagecache writeback requests, but depending on the
>> filesystem and storage hardware, data added to the repository may not be
>> @@ -680,8 +666,8 @@ backed up by any standard (e.g. POSIX), but worked in practice on some
>> Linux setups.
>> +
>> Nowadays you should almost certainly want to use
>> -`core.fsync=loose-object` instead in combination with
>> -`core.fsyncMethod=bulk`, and possibly with
>> +`core.fsync=true` instead in combination with
>> +`core.fsyncMethod=batch`, and possibly with
>> `fsyncMethod.batch.quarantine=true`, see above. On modern OS's (Linux,
>> OSX, Windows) that gives you most of the performance benefit of
>> `core.fsyncObjectFiles=false` with all of the safety of the old
>
> I'm at the point where I don't want to endlessly revisit this discussion.
Sorry, my intention isn't to frustrate you, but I do think it's
important to get this right.
Particularly since this is now in "next", and we're getting closer to a
release. We can either talk about this now and decide on something, or
it'll be in a release, and then publicly documented promises will be
harder to back out of.
I think your suggestion of just hiding the relevant documentation would
be a good band-aid solution to that.
But I also think that given how I was altering this in my RFC series
that the premise of how this could be structured has been called into
question in a way that we didn't (or I don't recall) us having discussed
before.
I.e. that we can say "sync loose, but not index", or "sync index, but
not loose" with this config schema. When with "bulk" we it really isn't
any more expensive to do both if one is true (even cheaper, actually).
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: do we have too much fsync() configuration in 'next'?
2022-03-26 0:24 ` Ævar Arnfjörð Bjarmason
@ 2022-03-26 1:23 ` Junio C Hamano
2022-03-26 1:25 ` do we have too much fsync() configuration in 'next'? (was: [PATCH v7] core.fsync: documentation and user-friendly aggregate options) Neeraj Singh
1 sibling, 0 replies; 122+ messages in thread
From: Junio C Hamano @ 2022-03-26 1:23 UTC (permalink / raw)
To: Ævar Arnfjörð Bjarmason
Cc: Neeraj Singh, Neeraj K. Singh via GitGitGadget,
Johannes Schindelin, Bagas Sanjaya, Git List, Neeraj Singh,
Elijah Newren, Patrick Steinhardt, Randall S. Becker,
brian m. carlson
Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:
>> We've gone over this a few times already, but just wanted to state it
>> again. The really detailed settings are really there for Git hosters
>> like GitLab or GitHub. I'd be happy to remove the per-component
>> core.fsync values from the documentation and leave just the ones we
>> point the user to.
>
> I'm prettty sure (but Patrick knows more) that GitLab's plan for this is
> to keep it at whatever the safest setting is, presumably GitHub's as
> well (but I don't know at all on that front).
I thought we've already settled it long ago, and it wasn't like you
were taking vacation from the list for a few weeks. Why are we
talking about this again now? Did we discover any new argument
against it?
Puzzled.
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: do we have too much fsync() configuration in 'next'? (was: [PATCH v7] core.fsync: documentation and user-friendly aggregate options)
2022-03-26 0:24 ` Ævar Arnfjörð Bjarmason
2022-03-26 1:23 ` do we have too much fsync() configuration in 'next'? Junio C Hamano
@ 2022-03-26 1:25 ` Neeraj Singh
2022-03-26 15:31 ` Ævar Arnfjörð Bjarmason
1 sibling, 1 reply; 122+ messages in thread
From: Neeraj Singh @ 2022-03-26 1:25 UTC (permalink / raw)
To: Ævar Arnfjörð Bjarmason
Cc: Neeraj K. Singh via GitGitGadget, Johannes Schindelin,
Bagas Sanjaya, Git List, Neeraj Singh, Elijah Newren,
Patrick Steinhardt, Randall S. Becker, brian m. carlson
On Fri, Mar 25, 2022 at 5:33 PM Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
>
>
> On Fri, Mar 25 2022, Neeraj Singh wrote:
>
> > On Wed, Mar 23, 2022 at 7:46 AM Ævar Arnfjörð Bjarmason
> > <avarab@gmail.com> wrote:
> >>
> >>
> >> On Tue, Mar 15 2022, Neeraj Singh wrote:
> >>
> >> I know this is probably 80% my fault by egging you on about initially
> >> adding the wildmatch() based thing you didn't go for.
> >>
> >> But having looked at this with fresh eyes quite deeply I really think
> >> we're severely over-configuring things here:
> >>
> >> > +core.fsync::
> >> > + A comma-separated list of components of the repository that
> >> > + should be hardened via the core.fsyncMethod when created or
> >> > + modified. You can disable hardening of any component by
> >> > + prefixing it with a '-'. Items that are not hardened may be
> >> > + lost in the event of an unclean system shutdown. Unless you
> >> > + have special requirements, it is recommended that you leave
> >> > + this option empty or pick one of `committed`, `added`,
> >> > + or `all`.
> >> > ++
> >> > +When this configuration is encountered, the set of components starts with
> >> > +the platform default value, disabled components are removed, and additional
> >> > +components are added. `none` resets the state so that the platform default
> >> > +is ignored.
> >> > ++
> >> > +The empty string resets the fsync configuration to the platform
> >> > +default. The default on most platforms is equivalent to
> >> > +`core.fsync=committed,-loose-object`, which has good performance,
> >> > +but risks losing recent work in the event of an unclean system shutdown.
> >> > ++
> >> > +* `none` clears the set of fsynced components.
> >> > +* `loose-object` hardens objects added to the repo in loose-object form.
> >> > +* `pack` hardens objects added to the repo in packfile form.
> >> > +* `pack-metadata` hardens packfile bitmaps and indexes.
> >> > +* `commit-graph` hardens the commit graph file.
> >> > +* `index` hardens the index when it is modified.
> >> > +* `objects` is an aggregate option that is equivalent to
> >> > + `loose-object,pack`.
> >> > +* `derived-metadata` is an aggregate option that is equivalent to
> >> > + `pack-metadata,commit-graph`.
> >> > +* `committed` is an aggregate option that is currently equivalent to
> >> > + `objects`. This mode sacrifices some performance to ensure that work
> >> > + that is committed to the repository with `git commit` or similar commands
> >> > + is hardened.
> >> > +* `added` is an aggregate option that is currently equivalent to
> >> > + `committed,index`. This mode sacrifices additional performance to
> >> > + ensure that the results of commands like `git add` and similar operations
> >> > + are hardened.
> >> > +* `all` is an aggregate option that syncs all individual components above.
> >> > +
> >> > core.fsyncMethod::
> >> > A value indicating the strategy Git will use to harden repository data
> >> > using fsync and related primitives.
> >>
> >> On top of my
> >> https://lore.kernel.org/git/RFC-patch-v2-7.7-a5951366c6e-20220323T140753Z-avarab@gmail.com/
> >> which makes the tmp-objdir part of your not-in-next-just-seen follow-up
> >> series configurable via "fsyncMethod.batch.quarantine" I really think we
> >> should just go for something like the belwo patch (note that
> >> misspelled/mistook "bulk" for "batch" in that linked-t patch, fixed
> >> below.
> >>
> >> I.e. I think we should just do our default fsync() of everything, and
> >> probably SOON make the fsync-ing of loose objects the default. Those who
> >> care about performance will have "batch" (or "writeout-only"), which we
> >> can have OS-specific detections for.
> >>
> >> But really, all of the rest of this is unduly boxing us into
> >> overconfiguration that I think nobody really needs.
> >>
> >
> > We've gone over this a few times already, but just wanted to state it
> > again. The really detailed settings are really there for Git hosters
> > like GitLab or GitHub. I'd be happy to remove the per-component
> > core.fsync values from the documentation and leave just the ones we
> > point the user to.
>
> I'm prettty sure (but Patrick knows more) that GitLab's plan for this is
> to keep it at whatever the safest setting is, presumably GitHub's as
> well (but I don't know at all on that front).
>
> >> If someone really needs this level of detail they can LD_PRELOAD
> >> something to have fsync intercept fd's and paths, and act appropriately.
> >>
> >> Worse, as the RFC series I sent
> >> (https://lore.kernel.org/git/RFC-cover-v2-0.7-00000000000-20220323T140753Z-avarab@gmail.com/)
> >> shows we can and should "batch" up fsync() operations across these
> >> configuration boundaries, which this level of configuration would seem
> >> to preclude.
> >>
> >> Or, we'd need to explain why "core.fsync=loose-object" won't *actually*
> >> call fsync() on a single loose object's fd under "batch" as I had to do
> >> on top of this in
> >> https://lore.kernel.org/git/RFC-patch-v2-6.7-c20301d7967-20220323T140753Z-avarab@gmail.com/
> >>
> >
> > 99.9% of users don't care and won't look. The ones who do look deeper
> > and understand the issues have source code and access to this ML
> > discussion to understand why this works this way.
>
> Exactly, so we can hopefully have a simpler interface.
>
> >> The same is going to apply for almost all of the rest of these
> >> configuration categories.
> >>
> >> I.e. a natural follow-up to e.g. batching across objects & index as I'm
> >> doing in
> >> https://lore.kernel.org/git/RFC-patch-v2-4.7-61f4f3d7ef4-20220323T140753Z-avarab@gmail.com/
> >> is to do likewise for all the PACK-related stuff before we rename it
> >> in-place. Or even have "git gc" issue only a single fsync() for all of
> >> PACKs, their metadata files, commit-graph etc., and then rename() things
> >> in-place as appropriate afterwards.
> >>
> >> diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
> >> index 365a12dc7ae..536238e209b 100644
> >> --- a/Documentation/config/core.txt
> >> +++ b/Documentation/config/core.txt
> >> @@ -548,49 +548,35 @@ core.whitespace::
> >> errors. The default tab width is 8. Allowed values are 1 to 63.
> >>
> >> core.fsync::
> >> - A comma-separated list of components of the repository that
> >> - should be hardened via the core.fsyncMethod when created or
> >> - modified. You can disable hardening of any component by
> >> - prefixing it with a '-'. Items that are not hardened may be
> >> - lost in the event of an unclean system shutdown. Unless you
> >> - have special requirements, it is recommended that you leave
> >> - this option empty or pick one of `committed`, `added`,
> >> - or `all`.
> >> -+
> >> -When this configuration is encountered, the set of components starts with
> >> -the platform default value, disabled components are removed, and additional
> >> -components are added. `none` resets the state so that the platform default
> >> -is ignored.
> >> -+
> >> -The empty string resets the fsync configuration to the platform
> >> -default. The default on most platforms is equivalent to
> >> -`core.fsync=committed,-loose-object`, which has good performance,
> >> -but risks losing recent work in the event of an unclean system shutdown.
> >> -+
> >> -* `none` clears the set of fsynced components.
> >> -* `loose-object` hardens objects added to the repo in loose-object form.
> >> -* `pack` hardens objects added to the repo in packfile form.
> >> -* `pack-metadata` hardens packfile bitmaps and indexes.
> >> -* `commit-graph` hardens the commit graph file.
> >> -* `index` hardens the index when it is modified.
> >> -* `objects` is an aggregate option that is equivalent to
> >> - `loose-object,pack`.
> >> -* `derived-metadata` is an aggregate option that is equivalent to
> >> - `pack-metadata,commit-graph`.
> >> -* `committed` is an aggregate option that is currently equivalent to
> >> - `objects`. This mode sacrifices some performance to ensure that work
> >> - that is committed to the repository with `git commit` or similar commands
> >> - is hardened.
> >> -* `added` is an aggregate option that is currently equivalent to
> >> - `committed,index`. This mode sacrifices additional performance to
> >> - ensure that the results of commands like `git add` and similar operations
> >> - are hardened.
> >> -* `all` is an aggregate option that syncs all individual components above.
> >> + A boolen defaulting to `true`. To ensure data integrity git
> >> + will fsync() its objects, index and refu updates etc. This can
> >> + be set to `false` to disable `fsync()`-ing.
> >> ++
> >> +Only set this to `false` if you know what you're doing, and are
> >> +prepared to deal with data corruption. Valid use-cases include
> >> +throwaway uses of repositories on ramdisks, one-off mass-imports
> >> +followed by calling `sync(1)` etc.
> >> ++
> >> +Note that the syncing of loose objects is currently excluded from
> >> +`core.fsync=true`. To turn on all fsync-ing you'll need
> >> +`core.fsync=true` and `core.fsyncObjectFiles=true`, but see
> >> +`core.fsyncMethod=batch` below for a much faster alternative that's
> >> +just as safe on various modern OS's.
> >> ++
> >> +The default is in flux and may change in the future, in particular the
> >> +equivalent of the already-deprecated `core.fsyncObjectFiles` setting
> >> +might soon default to `true`, and `core.fsyncMethod`'s default of
> >> +`fsync` might default to a setting deemed to be safe on the local OS,
> >> +suc has `batch` or `writeout-only`
> >>
> >> core.fsyncMethod::
> >> A value indicating the strategy Git will use to harden repository data
> >> using fsync and related primitives.
> >> +
> >> +Defaults to `fsync`, but as discussed for `core.fsync` above might
> >> +change to use one of the values below taking advantage of
> >> +platform-specific "faster `fsync()`".
> >> ++
> >> * `fsync` uses the fsync() system call or platform equivalents.
> >> * `writeout-only` issues pagecache writeback requests, but depending on the
> >> filesystem and storage hardware, data added to the repository may not be
> >> @@ -680,8 +666,8 @@ backed up by any standard (e.g. POSIX), but worked in practice on some
> >> Linux setups.
> >> +
> >> Nowadays you should almost certainly want to use
> >> -`core.fsync=loose-object` instead in combination with
> >> -`core.fsyncMethod=bulk`, and possibly with
> >> +`core.fsync=true` instead in combination with
> >> +`core.fsyncMethod=batch`, and possibly with
> >> `fsyncMethod.batch.quarantine=true`, see above. On modern OS's (Linux,
> >> OSX, Windows) that gives you most of the performance benefit of
> >> `core.fsyncObjectFiles=false` with all of the safety of the old
> >
> > I'm at the point where I don't want to endlessly revisit this discussion.
>
> Sorry, my intention isn't to frustrate you, but I do think it's
> important to get this right.
>
> Particularly since this is now in "next", and we're getting closer to a
> release. We can either talk about this now and decide on something, or
> it'll be in a release, and then publicly documented promises will be
> harder to back out of.
>
> I think your suggestion of just hiding the relevant documentation would
> be a good band-aid solution to that.
>
> But I also think that given how I was altering this in my RFC series
> that the premise of how this could be structured has been called into
> question in a way that we didn't (or I don't recall) us having discussed
> before.
>
> I.e. that we can say "sync loose, but not index", or "sync index, but
> not loose" with this config schema. When with "bulk" we it really isn't
> any more expensive to do both if one is true (even cheaper, actually).
>
I want to make a comment about the Index here. Syncing the index is
strictly required for the "added" level of consistency, so that we
don't lose stuff that leaves the work tree but is staged. But my
Windows enlistment has an index that's 266MB, which would be painful
to sync even with all the optimizations. Maybe with split-index, this
wouldn't be so bad, but I just wanted to call out that some advanced
users may really care about the configurability.
As Git's various database implementations improve, the fsync stuff
will hopefully be more optimal and self-tuning. But as that happens,
Git could just start ignoring settings that lose meaning without tying
anyones hands.
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: do we have too much fsync() configuration in 'next'? (was: [PATCH v7] core.fsync: documentation and user-friendly aggregate options)
2022-03-26 1:25 ` do we have too much fsync() configuration in 'next'? (was: [PATCH v7] core.fsync: documentation and user-friendly aggregate options) Neeraj Singh
@ 2022-03-26 15:31 ` Ævar Arnfjörð Bjarmason
2022-03-27 5:27 ` Neeraj Singh
0 siblings, 1 reply; 122+ messages in thread
From: Ævar Arnfjörð Bjarmason @ 2022-03-26 15:31 UTC (permalink / raw)
To: Neeraj Singh
Cc: Neeraj K. Singh via GitGitGadget, Johannes Schindelin,
Bagas Sanjaya, Git List, Neeraj Singh, Elijah Newren,
Patrick Steinhardt, Randall S. Becker, brian m. carlson
On Fri, Mar 25 2022, Neeraj Singh wrote:
> On Fri, Mar 25, 2022 at 5:33 PM Ævar Arnfjörð Bjarmason
> <avarab@gmail.com> wrote:
>>
>>
>> On Fri, Mar 25 2022, Neeraj Singh wrote:
>>
>> > On Wed, Mar 23, 2022 at 7:46 AM Ævar Arnfjörð Bjarmason
>> > <avarab@gmail.com> wrote:
>> >>
>> >>
>> >> On Tue, Mar 15 2022, Neeraj Singh wrote:
>> >>
>> >> I know this is probably 80% my fault by egging you on about initially
>> >> adding the wildmatch() based thing you didn't go for.
>> >>
>> >> But having looked at this with fresh eyes quite deeply I really think
>> >> we're severely over-configuring things here:
>> >>
>> >> > +core.fsync::
>> >> > + A comma-separated list of components of the repository that
>> >> > + should be hardened via the core.fsyncMethod when created or
>> >> > + modified. You can disable hardening of any component by
>> >> > + prefixing it with a '-'. Items that are not hardened may be
>> >> > + lost in the event of an unclean system shutdown. Unless you
>> >> > + have special requirements, it is recommended that you leave
>> >> > + this option empty or pick one of `committed`, `added`,
>> >> > + or `all`.
>> >> > ++
>> >> > +When this configuration is encountered, the set of components starts with
>> >> > +the platform default value, disabled components are removed, and additional
>> >> > +components are added. `none` resets the state so that the platform default
>> >> > +is ignored.
>> >> > ++
>> >> > +The empty string resets the fsync configuration to the platform
>> >> > +default. The default on most platforms is equivalent to
>> >> > +`core.fsync=committed,-loose-object`, which has good performance,
>> >> > +but risks losing recent work in the event of an unclean system shutdown.
>> >> > ++
>> >> > +* `none` clears the set of fsynced components.
>> >> > +* `loose-object` hardens objects added to the repo in loose-object form.
>> >> > +* `pack` hardens objects added to the repo in packfile form.
>> >> > +* `pack-metadata` hardens packfile bitmaps and indexes.
>> >> > +* `commit-graph` hardens the commit graph file.
>> >> > +* `index` hardens the index when it is modified.
>> >> > +* `objects` is an aggregate option that is equivalent to
>> >> > + `loose-object,pack`.
>> >> > +* `derived-metadata` is an aggregate option that is equivalent to
>> >> > + `pack-metadata,commit-graph`.
>> >> > +* `committed` is an aggregate option that is currently equivalent to
>> >> > + `objects`. This mode sacrifices some performance to ensure that work
>> >> > + that is committed to the repository with `git commit` or similar commands
>> >> > + is hardened.
>> >> > +* `added` is an aggregate option that is currently equivalent to
>> >> > + `committed,index`. This mode sacrifices additional performance to
>> >> > + ensure that the results of commands like `git add` and similar operations
>> >> > + are hardened.
>> >> > +* `all` is an aggregate option that syncs all individual components above.
>> >> > +
>> >> > core.fsyncMethod::
>> >> > A value indicating the strategy Git will use to harden repository data
>> >> > using fsync and related primitives.
>> >>
>> >> On top of my
>> >> https://lore.kernel.org/git/RFC-patch-v2-7.7-a5951366c6e-20220323T140753Z-avarab@gmail.com/
>> >> which makes the tmp-objdir part of your not-in-next-just-seen follow-up
>> >> series configurable via "fsyncMethod.batch.quarantine" I really think we
>> >> should just go for something like the belwo patch (note that
>> >> misspelled/mistook "bulk" for "batch" in that linked-t patch, fixed
>> >> below.
>> >>
>> >> I.e. I think we should just do our default fsync() of everything, and
>> >> probably SOON make the fsync-ing of loose objects the default. Those who
>> >> care about performance will have "batch" (or "writeout-only"), which we
>> >> can have OS-specific detections for.
>> >>
>> >> But really, all of the rest of this is unduly boxing us into
>> >> overconfiguration that I think nobody really needs.
>> >>
>> >
>> > We've gone over this a few times already, but just wanted to state it
>> > again. The really detailed settings are really there for Git hosters
>> > like GitLab or GitHub. I'd be happy to remove the per-component
>> > core.fsync values from the documentation and leave just the ones we
>> > point the user to.
>>
>> I'm prettty sure (but Patrick knows more) that GitLab's plan for this is
>> to keep it at whatever the safest setting is, presumably GitHub's as
>> well (but I don't know at all on that front).
>>
>> >> If someone really needs this level of detail they can LD_PRELOAD
>> >> something to have fsync intercept fd's and paths, and act appropriately.
>> >>
>> >> Worse, as the RFC series I sent
>> >> (https://lore.kernel.org/git/RFC-cover-v2-0.7-00000000000-20220323T140753Z-avarab@gmail.com/)
>> >> shows we can and should "batch" up fsync() operations across these
>> >> configuration boundaries, which this level of configuration would seem
>> >> to preclude.
>> >>
>> >> Or, we'd need to explain why "core.fsync=loose-object" won't *actually*
>> >> call fsync() on a single loose object's fd under "batch" as I had to do
>> >> on top of this in
>> >> https://lore.kernel.org/git/RFC-patch-v2-6.7-c20301d7967-20220323T140753Z-avarab@gmail.com/
>> >>
>> >
>> > 99.9% of users don't care and won't look. The ones who do look deeper
>> > and understand the issues have source code and access to this ML
>> > discussion to understand why this works this way.
>>
>> Exactly, so we can hopefully have a simpler interface.
>>
>> >> The same is going to apply for almost all of the rest of these
>> >> configuration categories.
>> >>
>> >> I.e. a natural follow-up to e.g. batching across objects & index as I'm
>> >> doing in
>> >> https://lore.kernel.org/git/RFC-patch-v2-4.7-61f4f3d7ef4-20220323T140753Z-avarab@gmail.com/
>> >> is to do likewise for all the PACK-related stuff before we rename it
>> >> in-place. Or even have "git gc" issue only a single fsync() for all of
>> >> PACKs, their metadata files, commit-graph etc., and then rename() things
>> >> in-place as appropriate afterwards.
>> >>
>> >> diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
>> >> index 365a12dc7ae..536238e209b 100644
>> >> --- a/Documentation/config/core.txt
>> >> +++ b/Documentation/config/core.txt
>> >> @@ -548,49 +548,35 @@ core.whitespace::
>> >> errors. The default tab width is 8. Allowed values are 1 to 63.
>> >>
>> >> core.fsync::
>> >> - A comma-separated list of components of the repository that
>> >> - should be hardened via the core.fsyncMethod when created or
>> >> - modified. You can disable hardening of any component by
>> >> - prefixing it with a '-'. Items that are not hardened may be
>> >> - lost in the event of an unclean system shutdown. Unless you
>> >> - have special requirements, it is recommended that you leave
>> >> - this option empty or pick one of `committed`, `added`,
>> >> - or `all`.
>> >> -+
>> >> -When this configuration is encountered, the set of components starts with
>> >> -the platform default value, disabled components are removed, and additional
>> >> -components are added. `none` resets the state so that the platform default
>> >> -is ignored.
>> >> -+
>> >> -The empty string resets the fsync configuration to the platform
>> >> -default. The default on most platforms is equivalent to
>> >> -`core.fsync=committed,-loose-object`, which has good performance,
>> >> -but risks losing recent work in the event of an unclean system shutdown.
>> >> -+
>> >> -* `none` clears the set of fsynced components.
>> >> -* `loose-object` hardens objects added to the repo in loose-object form.
>> >> -* `pack` hardens objects added to the repo in packfile form.
>> >> -* `pack-metadata` hardens packfile bitmaps and indexes.
>> >> -* `commit-graph` hardens the commit graph file.
>> >> -* `index` hardens the index when it is modified.
>> >> -* `objects` is an aggregate option that is equivalent to
>> >> - `loose-object,pack`.
>> >> -* `derived-metadata` is an aggregate option that is equivalent to
>> >> - `pack-metadata,commit-graph`.
>> >> -* `committed` is an aggregate option that is currently equivalent to
>> >> - `objects`. This mode sacrifices some performance to ensure that work
>> >> - that is committed to the repository with `git commit` or similar commands
>> >> - is hardened.
>> >> -* `added` is an aggregate option that is currently equivalent to
>> >> - `committed,index`. This mode sacrifices additional performance to
>> >> - ensure that the results of commands like `git add` and similar operations
>> >> - are hardened.
>> >> -* `all` is an aggregate option that syncs all individual components above.
>> >> + A boolen defaulting to `true`. To ensure data integrity git
>> >> + will fsync() its objects, index and refu updates etc. This can
>> >> + be set to `false` to disable `fsync()`-ing.
>> >> ++
>> >> +Only set this to `false` if you know what you're doing, and are
>> >> +prepared to deal with data corruption. Valid use-cases include
>> >> +throwaway uses of repositories on ramdisks, one-off mass-imports
>> >> +followed by calling `sync(1)` etc.
>> >> ++
>> >> +Note that the syncing of loose objects is currently excluded from
>> >> +`core.fsync=true`. To turn on all fsync-ing you'll need
>> >> +`core.fsync=true` and `core.fsyncObjectFiles=true`, but see
>> >> +`core.fsyncMethod=batch` below for a much faster alternative that's
>> >> +just as safe on various modern OS's.
>> >> ++
>> >> +The default is in flux and may change in the future, in particular the
>> >> +equivalent of the already-deprecated `core.fsyncObjectFiles` setting
>> >> +might soon default to `true`, and `core.fsyncMethod`'s default of
>> >> +`fsync` might default to a setting deemed to be safe on the local OS,
>> >> +suc has `batch` or `writeout-only`
>> >>
>> >> core.fsyncMethod::
>> >> A value indicating the strategy Git will use to harden repository data
>> >> using fsync and related primitives.
>> >> +
>> >> +Defaults to `fsync`, but as discussed for `core.fsync` above might
>> >> +change to use one of the values below taking advantage of
>> >> +platform-specific "faster `fsync()`".
>> >> ++
>> >> * `fsync` uses the fsync() system call or platform equivalents.
>> >> * `writeout-only` issues pagecache writeback requests, but depending on the
>> >> filesystem and storage hardware, data added to the repository may not be
>> >> @@ -680,8 +666,8 @@ backed up by any standard (e.g. POSIX), but worked in practice on some
>> >> Linux setups.
>> >> +
>> >> Nowadays you should almost certainly want to use
>> >> -`core.fsync=loose-object` instead in combination with
>> >> -`core.fsyncMethod=bulk`, and possibly with
>> >> +`core.fsync=true` instead in combination with
>> >> +`core.fsyncMethod=batch`, and possibly with
>> >> `fsyncMethod.batch.quarantine=true`, see above. On modern OS's (Linux,
>> >> OSX, Windows) that gives you most of the performance benefit of
>> >> `core.fsyncObjectFiles=false` with all of the safety of the old
>> >
>> > I'm at the point where I don't want to endlessly revisit this discussion.
>>
>> Sorry, my intention isn't to frustrate you, but I do think it's
>> important to get this right.
>>
>> Particularly since this is now in "next", and we're getting closer to a
>> release. We can either talk about this now and decide on something, or
>> it'll be in a release, and then publicly documented promises will be
>> harder to back out of.
>>
>> I think your suggestion of just hiding the relevant documentation would
>> be a good band-aid solution to that.
>>
>> But I also think that given how I was altering this in my RFC series
>> that the premise of how this could be structured has been called into
>> question in a way that we didn't (or I don't recall) us having discussed
>> before.
>>
>> I.e. that we can say "sync loose, but not index", or "sync index, but
>> not loose" with this config schema. When with "bulk" we it really isn't
>> any more expensive to do both if one is true (even cheaper, actually).
>>
>
> I want to make a comment about the Index here. Syncing the index is
> strictly required for the "added" level of consistency, so that we
> don't lose stuff that leaves the work tree but is staged. But my
> Windows enlistment has an index that's 266MB, which would be painful
> to sync even with all the optimizations. Maybe with split-index, this
> wouldn't be so bad, but I just wanted to call out that some advanced
> users may really care about the configurability.
So for that use-case you'd like to fsync the loose objects (if any), but
not the index? So the FS will "flush" up to the index, and then queue
the index for later syncing to platter?
But even in that case don't the settings need to be tied to one another,
because in the method=bulk sync=index && sync=!loose case wouldn't we be
syncing "loose" in any case?
> As Git's various database implementations improve, the fsync stuff
> will hopefully be more optimal and self-tuning. But as that happens,
> Git could just start ignoring settings that lose meaning without tying
> anyones hands.
Yeah that would alleviate most of my concerns here, but the docs aren't
saying anything like that. Since you added them & they just landed, do
you mind doing a small follow-up where we e.g. say that these new
settings are "EXPERIMENTAL" or whatever, and subject to drastic change?
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: do we have too much fsync() configuration in 'next'? (was: [PATCH v7] core.fsync: documentation and user-friendly aggregate options)
2022-03-26 15:31 ` Ævar Arnfjörð Bjarmason
@ 2022-03-27 5:27 ` Neeraj Singh
2022-03-27 12:43 ` Ævar Arnfjörð Bjarmason
0 siblings, 1 reply; 122+ messages in thread
From: Neeraj Singh @ 2022-03-27 5:27 UTC (permalink / raw)
To: Ævar Arnfjörð Bjarmason
Cc: Neeraj K. Singh via GitGitGadget, Johannes Schindelin,
Bagas Sanjaya, Git List, Neeraj Singh, Elijah Newren,
Patrick Steinhardt, Randall S. Becker, brian m. carlson
On Sat, Mar 26, 2022 at 8:34 AM Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
>
>
> On Fri, Mar 25 2022, Neeraj Singh wrote:
>
> > On Fri, Mar 25, 2022 at 5:33 PM Ævar Arnfjörð Bjarmason
> > <avarab@gmail.com> wrote:
> >>
> >>
> >> On Fri, Mar 25 2022, Neeraj Singh wrote:
> >>
> >> > On Wed, Mar 23, 2022 at 7:46 AM Ævar Arnfjörð Bjarmason
> >> > <avarab@gmail.com> wrote:
> >> >>
> >> >>
> >> >> On Tue, Mar 15 2022, Neeraj Singh wrote:
> >> >>
> >> >> I know this is probably 80% my fault by egging you on about initially
> >> >> adding the wildmatch() based thing you didn't go for.
> >> >>
> >> >> But having looked at this with fresh eyes quite deeply I really think
> >> >> we're severely over-configuring things here:
> >> >>
> >> >> > +core.fsync::
> >> >> > + A comma-separated list of components of the repository that
> >> >> > + should be hardened via the core.fsyncMethod when created or
> >> >> > + modified. You can disable hardening of any component by
> >> >> > + prefixing it with a '-'. Items that are not hardened may be
> >> >> > + lost in the event of an unclean system shutdown. Unless you
> >> >> > + have special requirements, it is recommended that you leave
> >> >> > + this option empty or pick one of `committed`, `added`,
> >> >> > + or `all`.
> >> >> > ++
> >> >> > +When this configuration is encountered, the set of components starts with
> >> >> > +the platform default value, disabled components are removed, and additional
> >> >> > +components are added. `none` resets the state so that the platform default
> >> >> > +is ignored.
> >> >> > ++
> >> >> > +The empty string resets the fsync configuration to the platform
> >> >> > +default. The default on most platforms is equivalent to
> >> >> > +`core.fsync=committed,-loose-object`, which has good performance,
> >> >> > +but risks losing recent work in the event of an unclean system shutdown.
> >> >> > ++
> >> >> > +* `none` clears the set of fsynced components.
> >> >> > +* `loose-object` hardens objects added to the repo in loose-object form.
> >> >> > +* `pack` hardens objects added to the repo in packfile form.
> >> >> > +* `pack-metadata` hardens packfile bitmaps and indexes.
> >> >> > +* `commit-graph` hardens the commit graph file.
> >> >> > +* `index` hardens the index when it is modified.
> >> >> > +* `objects` is an aggregate option that is equivalent to
> >> >> > + `loose-object,pack`.
> >> >> > +* `derived-metadata` is an aggregate option that is equivalent to
> >> >> > + `pack-metadata,commit-graph`.
> >> >> > +* `committed` is an aggregate option that is currently equivalent to
> >> >> > + `objects`. This mode sacrifices some performance to ensure that work
> >> >> > + that is committed to the repository with `git commit` or similar commands
> >> >> > + is hardened.
> >> >> > +* `added` is an aggregate option that is currently equivalent to
> >> >> > + `committed,index`. This mode sacrifices additional performance to
> >> >> > + ensure that the results of commands like `git add` and similar operations
> >> >> > + are hardened.
> >> >> > +* `all` is an aggregate option that syncs all individual components above.
> >> >> > +
> >> >> > core.fsyncMethod::
> >> >> > A value indicating the strategy Git will use to harden repository data
> >> >> > using fsync and related primitives.
> >> >>
> >> >> On top of my
> >> >> https://lore.kernel.org/git/RFC-patch-v2-7.7-a5951366c6e-20220323T140753Z-avarab@gmail.com/
> >> >> which makes the tmp-objdir part of your not-in-next-just-seen follow-up
> >> >> series configurable via "fsyncMethod.batch.quarantine" I really think we
> >> >> should just go for something like the belwo patch (note that
> >> >> misspelled/mistook "bulk" for "batch" in that linked-t patch, fixed
> >> >> below.
> >> >>
> >> >> I.e. I think we should just do our default fsync() of everything, and
> >> >> probably SOON make the fsync-ing of loose objects the default. Those who
> >> >> care about performance will have "batch" (or "writeout-only"), which we
> >> >> can have OS-specific detections for.
> >> >>
> >> >> But really, all of the rest of this is unduly boxing us into
> >> >> overconfiguration that I think nobody really needs.
> >> >>
> >> >
> >> > We've gone over this a few times already, but just wanted to state it
> >> > again. The really detailed settings are really there for Git hosters
> >> > like GitLab or GitHub. I'd be happy to remove the per-component
> >> > core.fsync values from the documentation and leave just the ones we
> >> > point the user to.
> >>
> >> I'm prettty sure (but Patrick knows more) that GitLab's plan for this is
> >> to keep it at whatever the safest setting is, presumably GitHub's as
> >> well (but I don't know at all on that front).
> >>
> >> >> If someone really needs this level of detail they can LD_PRELOAD
> >> >> something to have fsync intercept fd's and paths, and act appropriately.
> >> >>
> >> >> Worse, as the RFC series I sent
> >> >> (https://lore.kernel.org/git/RFC-cover-v2-0.7-00000000000-20220323T140753Z-avarab@gmail.com/)
> >> >> shows we can and should "batch" up fsync() operations across these
> >> >> configuration boundaries, which this level of configuration would seem
> >> >> to preclude.
> >> >>
> >> >> Or, we'd need to explain why "core.fsync=loose-object" won't *actually*
> >> >> call fsync() on a single loose object's fd under "batch" as I had to do
> >> >> on top of this in
> >> >> https://lore.kernel.org/git/RFC-patch-v2-6.7-c20301d7967-20220323T140753Z-avarab@gmail.com/
> >> >>
> >> >
> >> > 99.9% of users don't care and won't look. The ones who do look deeper
> >> > and understand the issues have source code and access to this ML
> >> > discussion to understand why this works this way.
> >>
> >> Exactly, so we can hopefully have a simpler interface.
> >>
> >> >> The same is going to apply for almost all of the rest of these
> >> >> configuration categories.
> >> >>
> >> >> I.e. a natural follow-up to e.g. batching across objects & index as I'm
> >> >> doing in
> >> >> https://lore.kernel.org/git/RFC-patch-v2-4.7-61f4f3d7ef4-20220323T140753Z-avarab@gmail.com/
> >> >> is to do likewise for all the PACK-related stuff before we rename it
> >> >> in-place. Or even have "git gc" issue only a single fsync() for all of
> >> >> PACKs, their metadata files, commit-graph etc., and then rename() things
> >> >> in-place as appropriate afterwards.
> >> >>
> >> >> diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
> >> >> index 365a12dc7ae..536238e209b 100644
> >> >> --- a/Documentation/config/core.txt
> >> >> +++ b/Documentation/config/core.txt
> >> >> @@ -548,49 +548,35 @@ core.whitespace::
> >> >> errors. The default tab width is 8. Allowed values are 1 to 63.
> >> >>
> >> >> core.fsync::
> >> >> - A comma-separated list of components of the repository that
> >> >> - should be hardened via the core.fsyncMethod when created or
> >> >> - modified. You can disable hardening of any component by
> >> >> - prefixing it with a '-'. Items that are not hardened may be
> >> >> - lost in the event of an unclean system shutdown. Unless you
> >> >> - have special requirements, it is recommended that you leave
> >> >> - this option empty or pick one of `committed`, `added`,
> >> >> - or `all`.
> >> >> -+
> >> >> -When this configuration is encountered, the set of components starts with
> >> >> -the platform default value, disabled components are removed, and additional
> >> >> -components are added. `none` resets the state so that the platform default
> >> >> -is ignored.
> >> >> -+
> >> >> -The empty string resets the fsync configuration to the platform
> >> >> -default. The default on most platforms is equivalent to
> >> >> -`core.fsync=committed,-loose-object`, which has good performance,
> >> >> -but risks losing recent work in the event of an unclean system shutdown.
> >> >> -+
> >> >> -* `none` clears the set of fsynced components.
> >> >> -* `loose-object` hardens objects added to the repo in loose-object form.
> >> >> -* `pack` hardens objects added to the repo in packfile form.
> >> >> -* `pack-metadata` hardens packfile bitmaps and indexes.
> >> >> -* `commit-graph` hardens the commit graph file.
> >> >> -* `index` hardens the index when it is modified.
> >> >> -* `objects` is an aggregate option that is equivalent to
> >> >> - `loose-object,pack`.
> >> >> -* `derived-metadata` is an aggregate option that is equivalent to
> >> >> - `pack-metadata,commit-graph`.
> >> >> -* `committed` is an aggregate option that is currently equivalent to
> >> >> - `objects`. This mode sacrifices some performance to ensure that work
> >> >> - that is committed to the repository with `git commit` or similar commands
> >> >> - is hardened.
> >> >> -* `added` is an aggregate option that is currently equivalent to
> >> >> - `committed,index`. This mode sacrifices additional performance to
> >> >> - ensure that the results of commands like `git add` and similar operations
> >> >> - are hardened.
> >> >> -* `all` is an aggregate option that syncs all individual components above.
> >> >> + A boolen defaulting to `true`. To ensure data integrity git
> >> >> + will fsync() its objects, index and refu updates etc. This can
> >> >> + be set to `false` to disable `fsync()`-ing.
> >> >> ++
> >> >> +Only set this to `false` if you know what you're doing, and are
> >> >> +prepared to deal with data corruption. Valid use-cases include
> >> >> +throwaway uses of repositories on ramdisks, one-off mass-imports
> >> >> +followed by calling `sync(1)` etc.
> >> >> ++
> >> >> +Note that the syncing of loose objects is currently excluded from
> >> >> +`core.fsync=true`. To turn on all fsync-ing you'll need
> >> >> +`core.fsync=true` and `core.fsyncObjectFiles=true`, but see
> >> >> +`core.fsyncMethod=batch` below for a much faster alternative that's
> >> >> +just as safe on various modern OS's.
> >> >> ++
> >> >> +The default is in flux and may change in the future, in particular the
> >> >> +equivalent of the already-deprecated `core.fsyncObjectFiles` setting
> >> >> +might soon default to `true`, and `core.fsyncMethod`'s default of
> >> >> +`fsync` might default to a setting deemed to be safe on the local OS,
> >> >> +suc has `batch` or `writeout-only`
> >> >>
> >> >> core.fsyncMethod::
> >> >> A value indicating the strategy Git will use to harden repository data
> >> >> using fsync and related primitives.
> >> >> +
> >> >> +Defaults to `fsync`, but as discussed for `core.fsync` above might
> >> >> +change to use one of the values below taking advantage of
> >> >> +platform-specific "faster `fsync()`".
> >> >> ++
> >> >> * `fsync` uses the fsync() system call or platform equivalents.
> >> >> * `writeout-only` issues pagecache writeback requests, but depending on the
> >> >> filesystem and storage hardware, data added to the repository may not be
> >> >> @@ -680,8 +666,8 @@ backed up by any standard (e.g. POSIX), but worked in practice on some
> >> >> Linux setups.
> >> >> +
> >> >> Nowadays you should almost certainly want to use
> >> >> -`core.fsync=loose-object` instead in combination with
> >> >> -`core.fsyncMethod=bulk`, and possibly with
> >> >> +`core.fsync=true` instead in combination with
> >> >> +`core.fsyncMethod=batch`, and possibly with
> >> >> `fsyncMethod.batch.quarantine=true`, see above. On modern OS's (Linux,
> >> >> OSX, Windows) that gives you most of the performance benefit of
> >> >> `core.fsyncObjectFiles=false` with all of the safety of the old
> >> >
> >> > I'm at the point where I don't want to endlessly revisit this discussion.
> >>
> >> Sorry, my intention isn't to frustrate you, but I do think it's
> >> important to get this right.
> >>
> >> Particularly since this is now in "next", and we're getting closer to a
> >> release. We can either talk about this now and decide on something, or
> >> it'll be in a release, and then publicly documented promises will be
> >> harder to back out of.
> >>
> >> I think your suggestion of just hiding the relevant documentation would
> >> be a good band-aid solution to that.
> >>
> >> But I also think that given how I was altering this in my RFC series
> >> that the premise of how this could be structured has been called into
> >> question in a way that we didn't (or I don't recall) us having discussed
> >> before.
> >>
> >> I.e. that we can say "sync loose, but not index", or "sync index, but
> >> not loose" with this config schema. When with "bulk" we it really isn't
> >> any more expensive to do both if one is true (even cheaper, actually).
> >>
> >
> > I want to make a comment about the Index here. Syncing the index is
> > strictly required for the "added" level of consistency, so that we
> > don't lose stuff that leaves the work tree but is staged. But my
> > Windows enlistment has an index that's 266MB, which would be painful
> > to sync even with all the optimizations. Maybe with split-index, this
> > wouldn't be so bad, but I just wanted to call out that some advanced
> > users may really care about the configurability.
>
> So for that use-case you'd like to fsync the loose objects (if any), but
> not the index? So the FS will "flush" up to the index, and then queue
> the index for later syncing to platter?
>
>
> But even in that case don't the settings need to be tied to one another,
> because in the method=bulk sync=index && sync=!loose case wouldn't we be
> syncing "loose" in any case?
>
> > As Git's various database implementations improve, the fsync stuff
> > will hopefully be more optimal and self-tuning. But as that happens,
> > Git could just start ignoring settings that lose meaning without tying
> > anyones hands.
>
> Yeah that would alleviate most of my concerns here, but the docs aren't
> saying anything like that. Since you added them & they just landed, do
> you mind doing a small follow-up where we e.g. say that these new
> settings are "EXPERIMENTAL" or whatever, and subject to drastic change?
The doc is already pretty prescriptive. It has this line at the end
of the first paragraph:
"Unless you
have special requirements, it is recommended that you leave
this option empty or pick one of `committed`, `added`,
or `all`."
Those values are already designed to change as Git changes.
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: do we have too much fsync() configuration in 'next'? (was: [PATCH v7] core.fsync: documentation and user-friendly aggregate options)
2022-03-27 5:27 ` Neeraj Singh
@ 2022-03-27 12:43 ` Ævar Arnfjörð Bjarmason
2022-03-28 10:56 ` Patrick Steinhardt
0 siblings, 1 reply; 122+ messages in thread
From: Ævar Arnfjörð Bjarmason @ 2022-03-27 12:43 UTC (permalink / raw)
To: Neeraj Singh
Cc: Neeraj K. Singh via GitGitGadget, Johannes Schindelin,
Bagas Sanjaya, Git List, Neeraj Singh, Elijah Newren,
Patrick Steinhardt, Randall S. Becker, brian m. carlson
On Sat, Mar 26 2022, Neeraj Singh wrote:
> On Sat, Mar 26, 2022 at 8:34 AM Ævar Arnfjörð Bjarmason
> <avarab@gmail.com> wrote:
>>
>>
>> On Fri, Mar 25 2022, Neeraj Singh wrote:
>>
>> > On Fri, Mar 25, 2022 at 5:33 PM Ævar Arnfjörð Bjarmason
>> > <avarab@gmail.com> wrote:
[...]
>> > I want to make a comment about the Index here. Syncing the index is
>> > strictly required for the "added" level of consistency, so that we
>> > don't lose stuff that leaves the work tree but is staged. But my
>> > Windows enlistment has an index that's 266MB, which would be painful
>> > to sync even with all the optimizations. Maybe with split-index, this
>> > wouldn't be so bad, but I just wanted to call out that some advanced
>> > users may really care about the configurability.
>>
>> So for that use-case you'd like to fsync the loose objects (if any), but
>> not the index? So the FS will "flush" up to the index, and then queue
>> the index for later syncing to platter?
>>
>>
>> But even in that case don't the settings need to be tied to one another,
>> because in the method=bulk sync=index && sync=!loose case wouldn't we be
>> syncing "loose" in any case?
>>
>> > As Git's various database implementations improve, the fsync stuff
>> > will hopefully be more optimal and self-tuning. But as that happens,
>> > Git could just start ignoring settings that lose meaning without tying
>> > anyones hands.
>>
>> Yeah that would alleviate most of my concerns here, but the docs aren't
>> saying anything like that. Since you added them & they just landed, do
>> you mind doing a small follow-up where we e.g. say that these new
>> settings are "EXPERIMENTAL" or whatever, and subject to drastic change?
>
> The doc is already pretty prescriptive. It has this line at the end
> of the first paragraph:
> "Unless you
> have special requirements, it is recommended that you leave
> this option empty or pick one of `committed`, `added`,
> or `all`."
>
> Those values are already designed to change as Git changes.
I'm referring to the documentation as it stands not being marked as
experimental in the sense that we might decide to re-do this to a large
extent, i.e. something like the diff I suggested upthread in
https://lore.kernel.org/git/220323.86fsn8ohg8.gmgdl@evledraar.gmail.com/
So yes, I agree that it e.g. clearly states that you can add a new
core.git=foobar or whatever down the line, but it clearly doesn't
suggest that e.g. core.fsync might have boolean semantics in some later
version, or that the rest might simply be ignored, even if that
e.g. means that we wouldn't sync loose objects on
core.fsync=loose-object, as we'd just warn with a "we don't provide this
anymore".
Or do you disagree with that? IOW I mean that we'd do something like
this, either in docs or code:
diff --git a/config.c b/config.c
index 3c9b6b589ab..94548566073 100644
--- a/config.c
+++ b/config.c
@@ -1675,6 +1675,9 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
}
if (!strcmp(var, "core.fsync")) {
+ if (!the_repository->settings.feature_experimental)
+ warning(_("the '%s' configuration option is EXPERIMENTAL. opt-in to use it with feature.experimental=true"),
+ var);
if (!value)
return config_error_nonbool(var);
fsync_components = parse_fsync_components(var, value);
@@ -1682,6 +1685,9 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
}
if (!strcmp(var, "core.fsyncmethod")) {
+ if (!the_repository->settings.feature_experimental)
+ warning(_("the '%s' configuration option is EXPERIMENTAL. opt-in to use it with feature.experimental=true"),
+ var);
if (!value)
return config_error_nonbool(var);
if (!strcmp(value, "fsync"))
diff --git a/repo-settings.c b/repo-settings.c
index b4fbd16cdcc..f949b65b91e 100644
--- a/repo-settings.c
+++ b/repo-settings.c
@@ -31,6 +31,7 @@ void prepare_repo_settings(struct repository *r)
/* Booleans config or default, cascades to other settings */
repo_cfg_bool(r, "feature.manyfiles", &manyfiles, 0);
repo_cfg_bool(r, "feature.experimental", &experimental, 0);
+ r->settings.feature_experimental = experimental;
/* Defaults modified by feature.* */
if (experimental) {
diff --git a/repository.h b/repository.h
index e29f361703d..db8f99a8989 100644
--- a/repository.h
+++ b/repository.h
@@ -28,6 +28,7 @@ enum fetch_negotiation_setting {
struct repo_settings {
int initialized;
+ int feature_experimental;
int core_commit_graph;
int commit_graph_read_changed_paths;
int gc_write_commit_graph;
^ permalink raw reply related [flat|nested] 122+ messages in thread
* Re: do we have too much fsync() configuration in 'next'? (was: [PATCH v7] core.fsync: documentation and user-friendly aggregate options)
2022-03-27 12:43 ` Ævar Arnfjörð Bjarmason
@ 2022-03-28 10:56 ` Patrick Steinhardt
2022-03-28 11:25 ` Ævar Arnfjörð Bjarmason
0 siblings, 1 reply; 122+ messages in thread
From: Patrick Steinhardt @ 2022-03-28 10:56 UTC (permalink / raw)
To: Ævar Arnfjörð Bjarmason
Cc: Neeraj Singh, Neeraj K. Singh via GitGitGadget,
Johannes Schindelin, Bagas Sanjaya, Git List, Neeraj Singh,
Elijah Newren, Randall S. Becker, brian m. carlson
[-- Attachment #1: Type: text/plain, Size: 6443 bytes --]
On Sun, Mar 27, 2022 at 02:43:48PM +0200, Ævar Arnfjörð Bjarmason wrote:
>
> On Sat, Mar 26 2022, Neeraj Singh wrote:
>
> > On Sat, Mar 26, 2022 at 8:34 AM Ævar Arnfjörð Bjarmason
> > <avarab@gmail.com> wrote:
> >>
> >>
> >> On Fri, Mar 25 2022, Neeraj Singh wrote:
> >>
> >> > On Fri, Mar 25, 2022 at 5:33 PM Ævar Arnfjörð Bjarmason
> >> > <avarab@gmail.com> wrote:
> [...]
> >> > I want to make a comment about the Index here. Syncing the index is
> >> > strictly required for the "added" level of consistency, so that we
> >> > don't lose stuff that leaves the work tree but is staged. But my
> >> > Windows enlistment has an index that's 266MB, which would be painful
> >> > to sync even with all the optimizations. Maybe with split-index, this
> >> > wouldn't be so bad, but I just wanted to call out that some advanced
> >> > users may really care about the configurability.
> >>
> >> So for that use-case you'd like to fsync the loose objects (if any), but
> >> not the index? So the FS will "flush" up to the index, and then queue
> >> the index for later syncing to platter?
> >>
> >>
> >> But even in that case don't the settings need to be tied to one another,
> >> because in the method=bulk sync=index && sync=!loose case wouldn't we be
> >> syncing "loose" in any case?
> >>
> >> > As Git's various database implementations improve, the fsync stuff
> >> > will hopefully be more optimal and self-tuning. But as that happens,
> >> > Git could just start ignoring settings that lose meaning without tying
> >> > anyones hands.
> >>
> >> Yeah that would alleviate most of my concerns here, but the docs aren't
> >> saying anything like that. Since you added them & they just landed, do
> >> you mind doing a small follow-up where we e.g. say that these new
> >> settings are "EXPERIMENTAL" or whatever, and subject to drastic change?
> >
> > The doc is already pretty prescriptive. It has this line at the end
> > of the first paragraph:
> > "Unless you
> > have special requirements, it is recommended that you leave
> > this option empty or pick one of `committed`, `added`,
> > or `all`."
> >
> > Those values are already designed to change as Git changes.
>
> I'm referring to the documentation as it stands not being marked as
> experimental in the sense that we might decide to re-do this to a large
> extent, i.e. something like the diff I suggested upthread in
> https://lore.kernel.org/git/220323.86fsn8ohg8.gmgdl@evledraar.gmail.com/
>
> So yes, I agree that it e.g. clearly states that you can add a new
> core.git=foobar or whatever down the line, but it clearly doesn't
> suggest that e.g. core.fsync might have boolean semantics in some later
> version, or that the rest might simply be ignored, even if that
> e.g. means that we wouldn't sync loose objects on
> core.fsync=loose-object, as we'd just warn with a "we don't provide this
> anymore".
>
> Or do you disagree with that? IOW I mean that we'd do something like
> this, either in docs or code:
>
> diff --git a/config.c b/config.c
> index 3c9b6b589ab..94548566073 100644
> --- a/config.c
> +++ b/config.c
> @@ -1675,6 +1675,9 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
> }
>
> if (!strcmp(var, "core.fsync")) {
> + if (!the_repository->settings.feature_experimental)
> + warning(_("the '%s' configuration option is EXPERIMENTAL. opt-in to use it with feature.experimental=true"),
> + var);
> if (!value)
> return config_error_nonbool(var);
> fsync_components = parse_fsync_components(var, value);
> @@ -1682,6 +1685,9 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
> }
>
> if (!strcmp(var, "core.fsyncmethod")) {
> + if (!the_repository->settings.feature_experimental)
> + warning(_("the '%s' configuration option is EXPERIMENTAL. opt-in to use it with feature.experimental=true"),
> + var);
> if (!value)
> return config_error_nonbool(var);
> if (!strcmp(value, "fsync"))
Let's please not tie this to `feature.experimental=true`. Setting that
option has unintended sideeffects and will also change defaults which we
may not want to have in production. I don't mind adding a warning in the
docs though that the specific items which can be configured may be
subject to change in the future.
At GitLab, we've got a three-step plan:
1. We need to migrate to `core.fsync` in the first place. In order
to not migrate and change behaviour at the same point in time we
already benefit from the fine-grainedness of this config because
we can simply say `core.fsync=loose-objects` and have the same
behaviour as before with `core.fsyncLooseObjects=true`.
2. We'll want to enable syncing of packfiles, which I think wasn't
previously covered by `core.fsyncLooseobjects`.
3. We'll add `refs` to also sync loose refs to disk.
So while the end result will be the same as `committed`, having this
level of control helps us to assess the impact in a nicer way by being
able to do this step by step with feature flags.
On the other hand, many of the other parts we don't really care about.
Auxiliary metadata like the commit-graph or pack indices are data that
can in the worst case be regenerated by us, so it's not clear to me
whether it makes to also enable fsyncing those in production.
So altogether, I agree with Neeraj: having the fine-grainedness greatly
helps us to roll out changes like this and be able to pick what we deem
to be important. Personally I would be fine with explicitly pointing out
that there are two groups of this config in our docs though:
1. The "porcelain" group: "committed", "added", "all", "none". These
are abstract groups whose behaviour should adapt as we change
implementations, and are those that should typically be set by a
user, if intended.
2. The "plumbing" or "expert" group: these are fine-grained options
which shouldn't typically be used by Git users. They still have
merit though in hosting environments, where requirements are
typically a lot more specific.
We may also provide different guarantees for both groups. The first one
should definitely be stable, but we might state that the second group is
subject to change in the future.
Patrick
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v6 4/6] core.fsync: add configuration parsing
2022-03-10 22:43 ` [PATCH v6 4/6] core.fsync: add configuration parsing Neeraj Singh via GitGitGadget
@ 2022-03-28 11:06 ` Jiang Xin
2022-03-28 19:45 ` Neeraj Singh
0 siblings, 1 reply; 122+ messages in thread
From: Jiang Xin @ 2022-03-28 11:06 UTC (permalink / raw)
To: Neeraj Singh via GitGitGadget
Cc: Git List, rsbecker, Bagas Sanjaya, Elijah Newren,
Ævar Arnfjörð Bjarmason, nksingh85, ps,
brian m. carlson, Johannes Schindelin, Neeraj K. Singh
On Sat, Mar 12, 2022 at 6:25 AM Neeraj Singh via GitGitGadget
<gitgitgadget@gmail.com> wrote:
> @@ -1613,6 +1687,8 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
> }
>
> if (!strcmp(var, "core.fsyncobjectfiles")) {
> + if (fsync_object_files < 0)
> + warning(_("core.fsyncobjectfiles is deprecated; use core.fsync instead"));
s/core.fsyncobjectfiles/core.fsyncObjectFiles/ to use bumpyCaps for
config variable in documentation.
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: do we have too much fsync() configuration in 'next'? (was: [PATCH v7] core.fsync: documentation and user-friendly aggregate options)
2022-03-28 10:56 ` Patrick Steinhardt
@ 2022-03-28 11:25 ` Ævar Arnfjörð Bjarmason
2022-03-28 19:56 ` Neeraj Singh
0 siblings, 1 reply; 122+ messages in thread
From: Ævar Arnfjörð Bjarmason @ 2022-03-28 11:25 UTC (permalink / raw)
To: Patrick Steinhardt
Cc: Neeraj Singh, Neeraj K. Singh via GitGitGadget,
Johannes Schindelin, Bagas Sanjaya, Git List, Neeraj Singh,
Elijah Newren, Randall S. Becker, brian m. carlson,
'Eric Wong'
On Mon, Mar 28 2022, Patrick Steinhardt wrote:
> [[PGP Signed Part:Undecided]]
> On Sun, Mar 27, 2022 at 02:43:48PM +0200, Ævar Arnfjörð Bjarmason wrote:
>>
>> On Sat, Mar 26 2022, Neeraj Singh wrote:
>>
>> > On Sat, Mar 26, 2022 at 8:34 AM Ævar Arnfjörð Bjarmason
>> > <avarab@gmail.com> wrote:
>> >>
>> >>
>> >> On Fri, Mar 25 2022, Neeraj Singh wrote:
>> >>
>> >> > On Fri, Mar 25, 2022 at 5:33 PM Ævar Arnfjörð Bjarmason
>> >> > <avarab@gmail.com> wrote:
>> [...]
>> >> > I want to make a comment about the Index here. Syncing the index is
>> >> > strictly required for the "added" level of consistency, so that we
>> >> > don't lose stuff that leaves the work tree but is staged. But my
>> >> > Windows enlistment has an index that's 266MB, which would be painful
>> >> > to sync even with all the optimizations. Maybe with split-index, this
>> >> > wouldn't be so bad, but I just wanted to call out that some advanced
>> >> > users may really care about the configurability.
>> >>
>> >> So for that use-case you'd like to fsync the loose objects (if any), but
>> >> not the index? So the FS will "flush" up to the index, and then queue
>> >> the index for later syncing to platter?
>> >>
>> >>
>> >> But even in that case don't the settings need to be tied to one another,
>> >> because in the method=bulk sync=index && sync=!loose case wouldn't we be
>> >> syncing "loose" in any case?
>> >>
>> >> > As Git's various database implementations improve, the fsync stuff
>> >> > will hopefully be more optimal and self-tuning. But as that happens,
>> >> > Git could just start ignoring settings that lose meaning without tying
>> >> > anyones hands.
>> >>
>> >> Yeah that would alleviate most of my concerns here, but the docs aren't
>> >> saying anything like that. Since you added them & they just landed, do
>> >> you mind doing a small follow-up where we e.g. say that these new
>> >> settings are "EXPERIMENTAL" or whatever, and subject to drastic change?
>> >
>> > The doc is already pretty prescriptive. It has this line at the end
>> > of the first paragraph:
>> > "Unless you
>> > have special requirements, it is recommended that you leave
>> > this option empty or pick one of `committed`, `added`,
>> > or `all`."
>> >
>> > Those values are already designed to change as Git changes.
>>
>> I'm referring to the documentation as it stands not being marked as
>> experimental in the sense that we might decide to re-do this to a large
>> extent, i.e. something like the diff I suggested upthread in
>> https://lore.kernel.org/git/220323.86fsn8ohg8.gmgdl@evledraar.gmail.com/
>>
>> So yes, I agree that it e.g. clearly states that you can add a new
>> core.git=foobar or whatever down the line, but it clearly doesn't
>> suggest that e.g. core.fsync might have boolean semantics in some later
>> version, or that the rest might simply be ignored, even if that
>> e.g. means that we wouldn't sync loose objects on
>> core.fsync=loose-object, as we'd just warn with a "we don't provide this
>> anymore".
>>
>> Or do you disagree with that? IOW I mean that we'd do something like
>> this, either in docs or code:
>>
>> diff --git a/config.c b/config.c
>> index 3c9b6b589ab..94548566073 100644
>> --- a/config.c
>> +++ b/config.c
>> @@ -1675,6 +1675,9 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
>> }
>>
>> if (!strcmp(var, "core.fsync")) {
>> + if (!the_repository->settings.feature_experimental)
>> + warning(_("the '%s' configuration option is EXPERIMENTAL. opt-in to use it with feature.experimental=true"),
>> + var);
>> if (!value)
>> return config_error_nonbool(var);
>> fsync_components = parse_fsync_components(var, value);
>> @@ -1682,6 +1685,9 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
>> }
>>
>> if (!strcmp(var, "core.fsyncmethod")) {
>> + if (!the_repository->settings.feature_experimental)
>> + warning(_("the '%s' configuration option is EXPERIMENTAL. opt-in to use it with feature.experimental=true"),
>> + var);
>> if (!value)
>> return config_error_nonbool(var);
>> if (!strcmp(value, "fsync"))
>
> Let's please not tie this to `feature.experimental=true`. Setting that
> option has unintended sideeffects and will also change defaults which we
> may not want to have in production. I don't mind adding a warning in the
> docs though that the specific items which can be configured may be
> subject to change in the future.
Yes, that was a bad (throwaway) idea. I think probably any sort of
warning is over-doing it, but having the same in the docs would be good,
as in:
git gre EXPERIMENTAL -- Documentation
> At GitLab, we've got a three-step plan:
>
> 1. We need to migrate to `core.fsync` in the first place. In order
> to not migrate and change behaviour at the same point in time we
> already benefit from the fine-grainedness of this config because
> we can simply say `core.fsync=loose-objects` and have the same
> behaviour as before with `core.fsyncLooseObjects=true`.
*nod*.
> 2. We'll want to enable syncing of packfiles, which I think wasn't
> previously covered by `core.fsyncLooseobjects`.
We've always fsynced packfiles and other things that use the
finalize_hashfile() API. I.e. the pack metadata (idx,midx,bitmap etc.),
commit-graph etc.
Which is one thing I find a bit uncomfortable about the proposed config
schema, i.e. it's allowing *granular* unsafe behavior we didn't allow
before.
I think we *should* allow disabling fsync() entirely via:
core.fsync=false
Per Eric Wong's [added to CC] proposal here:
https://lore.kernel.org/git/20211028002102.19384-1-e@80x24.org/; I think
that's useful for e.g. running git in CI, one off scripted mass-imports
where you run "sync(1)" after (or not...).
But I don't really see the use-case for turning off say "index" or
"pack-metadata", but otherwise keeping the default fsync().
> 3. We'll add `refs` to also sync loose refs to disk.
Maybe I'm reading this wrong, but AFAICT if you upgrade from pre-v2.36.0
to v2.36.0 you'll have no way to do fsync()-ing as it was done before
with your bc22d845c43 (core.fsync: new option to harden references,
2022-03-11).
I.e. I first thought you meant to start with:
core.fsync=-loose-objects
core.fsync=-reference
And then remove that "core.fsync=-reference" line to get the behavior
that'll be new in v2.36.0, but that won't do that. The new "reference"
category doesn't just affect loose refs, but all ref updates.
So we don't have any way to move to v2.36.0 and get exactly the fsync()
behavior we did before, or have I misread the code?
Now, I don't think we need it to be configurable at all.
I.e. I think your is a bc22d845c43 good change, but it feels weird to
make it and leave the default of core.fsyncLooseObjects=false on the
table. I.e. what you're summarizing there is true, but it's also true of
the loose objects.
To the extent that we've had any reason at all not to sync those by
default (which has really mostly been "we didn't re-visit it for a
while") it's been performance.
And both loose refs & loose objects will suffer from the same
degradation in performance from many fsync()'s.
IOW I think it's perfectly fine not to add a config knob for it other
than core.fsync=false, and VERY SOON turn on
core.fsyncLooseobjects=true, especially if we can get most of the
performance benefits with the "bulk" mode.
But why half-way with bc22d845c43? I mean, *that change* should be
narrow, but in terms of where we go next what do you think of the above?
> So while the end result will be the same as `committed`, having this
> level of control helps us to assess the impact in a nicer way by being
> able to do this step by step with feature flags.
*Nod*, although leaving aside the new syncing of loose refs the plan you
outlined above could be done with the proposal of the simpler:
core.fsync=true
core.fsyncLooseObjects=false
> On the other hand, many of the other parts we don't really care about.
> Auxiliary metadata like the commit-graph or pack indices are data that
> can in the worst case be regenerated by us, so it's not clear to me
> whether it makes to also enable fsyncing those in production.
I'm not familiar with all of those in detail, i.e. how we behave
specifically in the face of them being corrupt.
The commit-graph I am, we *should* recover "gracefully" there, but you
might have an incident shortly there after due to "for-each-ref
--contains" slowdowns by 1000x or whatever.
For e.g. *.idx we'd be hosed until manual recovery.
So I think those area all in the same bucket as core.fsync=false, and
that we don't need the granularity.
> So altogether, I agree with Neeraj: having the fine-grainedness greatly
> helps us to roll out changes like this and be able to pick what we deem
> to be important. Personally I would be fine with explicitly pointing out
> that there are two groups of this config in our docs though:
Yes, that's fair. But pending replies to the above I think the main
point & proposal of us having too much config stands. I.e. depending on
what you want to do with loose object refs we'd just need this:
core.fsync=[<bool>] # true by default
core.fsyncLooseObjects=[<bool>] # false by default
core.fsyncLooseRefs=[<bool>] # true by default?
And then the "bulk" config, which would be orthagonal to this.
I.e. do we really have a use-case for the rest of the kitchen sink?
> 1. The "porcelain" group: "committed", "added", "all", "none". These
> are abstract groups whose behaviour should adapt as we change
> implementations, and are those that should typically be set by a
> user, if intended.
>
> 2. The "plumbing" or "expert" group: these are fine-grained options
> which shouldn't typically be used by Git users. They still have
> merit though in hosting environments, where requirements are
> typically a lot more specific.
>
> We may also provide different guarantees for both groups. The first one
> should definitely be stable, but we might state that the second group is
> subject to change in the future.
I hope we can work something out :)
Overall: I think you've left one of the the main things I brought up[1]
unaddressed, i.e. that the core.fsync config schema in its current form
assumes that we can sync A or B, and configure those separately.
Which AFAIKT is because Neeraj's initial implementation & the discussion
was focused on finishing A or B with a per-"group" "cookie" to flush the
files.
But as [2] shows it's more performant for us to simply defer the fsync
of A until the committing of B.
Which is the main reason I think we should be re-visiting this. Sure, if
we were just syncing A, B or C having per-[ABC] config options might be
a bit overdoing it, but would be relatively simple.
But once we start using a more optimized version of the "bulk" mode the
config schema will be making promises about individual steps in a
transaction that I think we'll want to leave opaque, and only promise
that when git returns it will have synced all the relevant assets as
efficiently as possible.
1. https://lore.kernel.org/git/220323.86fsn8ohg8.gmgdl@evledraar.gmail.com/
2. https://lore.kernel.org/git/RFC-patch-v2-4.7-61f4f3d7ef4-20220323T140753Z-avarab@gmail.com/
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v6 4/6] core.fsync: add configuration parsing
2022-03-28 11:06 ` Jiang Xin
@ 2022-03-28 19:45 ` Neeraj Singh
0 siblings, 0 replies; 122+ messages in thread
From: Neeraj Singh @ 2022-03-28 19:45 UTC (permalink / raw)
To: Jiang Xin
Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker,
Bagas Sanjaya, Elijah Newren,
Ævar Arnfjörð Bjarmason, Patrick Steinhardt,
brian m. carlson, Johannes Schindelin, Neeraj K. Singh
On Mon, Mar 28, 2022 at 4:07 AM Jiang Xin <worldhello.net@gmail.com> wrote:
>
> On Sat, Mar 12, 2022 at 6:25 AM Neeraj Singh via GitGitGadget
> <gitgitgadget@gmail.com> wrote:
> > @@ -1613,6 +1687,8 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
> > }
> >
> > if (!strcmp(var, "core.fsyncobjectfiles")) {
> > + if (fsync_object_files < 0)
> > + warning(_("core.fsyncobjectfiles is deprecated; use core.fsync instead"));
>
> s/core.fsyncobjectfiles/core.fsyncObjectFiles/ to use bumpyCaps for
> config variable in documentation.
THanks for pointing this out. I'll fix it.
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: do we have too much fsync() configuration in 'next'? (was: [PATCH v7] core.fsync: documentation and user-friendly aggregate options)
2022-03-28 11:25 ` Ævar Arnfjörð Bjarmason
@ 2022-03-28 19:56 ` Neeraj Singh
2022-03-30 16:59 ` Neeraj Singh
0 siblings, 1 reply; 122+ messages in thread
From: Neeraj Singh @ 2022-03-28 19:56 UTC (permalink / raw)
To: Ævar Arnfjörð Bjarmason
Cc: Patrick Steinhardt, Neeraj K. Singh via GitGitGadget,
Johannes Schindelin, Bagas Sanjaya, Git List, Neeraj Singh,
Elijah Newren, Randall S. Becker, brian m. carlson, Eric Wong
On Mon, Mar 28, 2022 at 5:15 AM Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
>
> I hope we can work something out :)
>
> Overall: I think you've left one of the the main things I brought up[1]
> unaddressed, i.e. that the core.fsync config schema in its current form
> assumes that we can sync A or B, and configure those separately.
>
> Which AFAIKT is because Neeraj's initial implementation & the discussion
> was focused on finishing A or B with a per-"group" "cookie" to flush the
> files.
>
> But as [2] shows it's more performant for us to simply defer the fsync
> of A until the committing of B.
>
> Which is the main reason I think we should be re-visiting this. Sure, if
> we were just syncing A, B or C having per-[ABC] config options might be
> a bit overdoing it, but would be relatively simple.
>
> But once we start using a more optimized version of the "bulk" mode the
> config schema will be making promises about individual steps in a
> transaction that I think we'll want to leave opaque, and only promise
> that when git returns it will have synced all the relevant assets as
> efficiently as possible.
>
> 1. https://lore.kernel.org/git/220323.86fsn8ohg8.gmgdl@evledraar.gmail.com/
> 2. https://lore.kernel.org/git/RFC-patch-v2-4.7-61f4f3d7ef4-20220323T140753Z-avarab@gmail.com/
I think the current documentation is fine (obviously since I wrote
it). Let's reproduce the first part again:
---
core.fsync::
A comma-separated list of components of the repository that
should be hardened via the core.fsyncMethod when created or
modified. You can disable hardening of any component by
prefixing it with a '-'. Items that are not hardened may be
lost in the event of an unclean system shutdown. Unless you
have special requirements, it is recommended that you leave
this option empty or pick one of `committed`, `added`,
or `all`.
+
When this configuration is encountered, the set of components starts with
the platform default value, disabled components are removed, and additional
components are added. `none` resets the state so that the platform default
is ignored.
+
The empty string resets the fsync configuration to the platform
default. The default on most platforms is equivalent to
`core.fsync=committed,-loose-object`, which has good performance,
but risks losing recent work in the event of an unclean system shutdown.
+
---
We're only talking about "hardening" parts of the repository, and we
say we'll do it using the "fsyncMethod". If you don't harden
something, you could lose it if the system dies. All of these
statements are true and don't say so much about the implementation of
how the hardening happens. It's perfectly valid to not force any
component out of the disk cache, and a straightforward implementation
of repo transactions can put the sync point anywhere. We also
explicitly point the user at a "porcelain" setting in the first
paragraph.
So I think you're alone in thinking that anything needs to change here.
Thanks,
Neeraj
^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: do we have too much fsync() configuration in 'next'? (was: [PATCH v7] core.fsync: documentation and user-friendly aggregate options)
2022-03-28 19:56 ` Neeraj Singh
@ 2022-03-30 16:59 ` Neeraj Singh
0 siblings, 0 replies; 122+ messages in thread
From: Neeraj Singh @ 2022-03-30 16:59 UTC (permalink / raw)
To: Ævar Arnfjörð Bjarmason
Cc: Patrick Steinhardt, Neeraj K. Singh via GitGitGadget,
Johannes Schindelin, Bagas Sanjaya, Git List, Neeraj Singh,
Elijah Newren, Randall S. Becker, brian m. carlson, Eric Wong
On Mon, Mar 28, 2022 at 12:56 PM Neeraj Singh <nksingh85@gmail.com> wrote:
>
> On Mon, Mar 28, 2022 at 5:15 AM Ævar Arnfjörð Bjarmason
> <avarab@gmail.com> wrote:
>
> So I think you're alone in thinking that anything needs to change here.
>
Ævar,
I apologize for the negative tone and content of this comment.
Thanks,
Neeraj
^ permalink raw reply [flat|nested] 122+ messages in thread
end of thread, other threads:[~2022-03-30 17:00 UTC | newest]
Thread overview: 122+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-12-04 3:28 [PATCH 0/2] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget
2021-12-04 3:28 ` [PATCH 1/2] fsync: add writeout-only mode for fsyncing repo data Neeraj Singh via GitGitGadget
2021-12-06 7:54 ` Neeraj Singh
2021-12-04 3:28 ` [PATCH 2/2] core.fsync: introduce granular fsync control Neeraj Singh via GitGitGadget
2021-12-07 2:46 ` [PATCH v2 0/3] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget
2021-12-07 2:46 ` [PATCH v2 1/3] core.fsyncmethod: add writeout-only mode Neeraj Singh via GitGitGadget
2021-12-07 11:44 ` Patrick Steinhardt
2021-12-07 12:14 ` Ævar Arnfjörð Bjarmason
2021-12-07 23:29 ` Neeraj Singh
2021-12-07 12:18 ` Ævar Arnfjörð Bjarmason
2021-12-07 23:58 ` Neeraj Singh
2021-12-07 2:46 ` [PATCH v2 2/3] core.fsync: introduce granular fsync control Neeraj Singh via GitGitGadget
2021-12-07 11:53 ` Patrick Steinhardt
2021-12-07 20:46 ` Neeraj Singh
2021-12-07 12:29 ` Ævar Arnfjörð Bjarmason
2021-12-07 21:44 ` Neeraj Singh
2021-12-08 10:05 ` Ævar Arnfjörð Bjarmason
2021-12-09 0:14 ` Neeraj Singh
2021-12-09 0:44 ` Junio C Hamano
2021-12-09 4:08 ` Ævar Arnfjörð Bjarmason
2021-12-09 6:18 ` Neeraj Singh
2022-01-18 23:50 ` Neeraj Singh
2022-01-19 15:28 ` Ævar Arnfjörð Bjarmason
2022-01-19 14:52 ` Ævar Arnfjörð Bjarmason
2022-01-28 1:28 ` Neeraj Singh
2021-12-07 2:46 ` [PATCH v2 3/3] core.fsync: new option to harden the index Neeraj Singh via GitGitGadget
2021-12-07 11:56 ` [PATCH v2 0/3] A design for future-proofing fsync() configuration Patrick Steinhardt
2021-12-08 0:44 ` Neeraj Singh
2021-12-09 0:57 ` [PATCH v3 0/4] " Neeraj K. Singh via GitGitGadget
2021-12-09 0:57 ` [PATCH v3 1/4] core.fsyncmethod: add writeout-only mode Neeraj Singh via GitGitGadget
2021-12-09 0:57 ` [PATCH v3 2/4] core.fsync: introduce granular fsync control Neeraj Singh via GitGitGadget
2021-12-09 0:57 ` [PATCH v3 3/4] core.fsync: new option to harden the index Neeraj Singh via GitGitGadget
2021-12-09 0:57 ` [PATCH v3 4/4] core.fsync: add a `derived-metadata` aggregate option Neeraj Singh via GitGitGadget
2022-01-08 1:13 ` [PATCH v3 0/4] A design for future-proofing fsync() configuration Neeraj Singh
2022-01-09 0:55 ` rsbecker
2022-01-10 19:00 ` Neeraj Singh
2022-02-01 3:33 ` [PATCH v4 " Neeraj K. Singh via GitGitGadget
2022-02-01 3:33 ` [PATCH v4 1/4] core.fsyncmethod: add writeout-only mode Neeraj Singh via GitGitGadget
2022-02-01 3:33 ` [PATCH v4 2/4] core.fsync: introduce granular fsync control Neeraj Singh via GitGitGadget
2022-02-02 0:51 ` Junio C Hamano
2022-02-02 1:42 ` Junio C Hamano
2022-02-11 21:18 ` Neeraj Singh
2022-02-11 22:19 ` Junio C Hamano
2022-02-11 23:04 ` Neeraj Singh
2022-02-11 23:15 ` Junio C Hamano
2022-02-12 0:39 ` rsbecker
2022-02-14 7:04 ` Patrick Steinhardt
2022-02-14 17:17 ` Junio C Hamano
2022-03-09 13:42 ` Patrick Steinhardt
2022-03-09 18:50 ` Ævar Arnfjörð Bjarmason
2022-03-09 20:03 ` Junio C Hamano
2022-03-10 12:33 ` Patrick Steinhardt
2022-03-10 17:15 ` Junio C Hamano
2022-03-09 20:05 ` Neeraj Singh
2022-02-11 20:38 ` Neeraj Singh
2022-02-01 3:33 ` [PATCH v4 3/4] core.fsync: new option to harden the index Neeraj Singh via GitGitGadget
2022-02-01 3:33 ` [PATCH v4 4/4] core.fsync: add a `derived-metadata` aggregate option Neeraj Singh via GitGitGadget
2022-03-09 23:03 ` [PATCH v5 0/5] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget
2022-03-09 23:03 ` [PATCH v5 1/5] wrapper: move inclusion of CSPRNG headers the wrapper.c file Neeraj Singh via GitGitGadget
2022-03-09 23:29 ` Junio C Hamano
2022-03-10 1:21 ` Neeraj Singh
2022-03-10 1:26 ` brian m. carlson
2022-03-10 1:56 ` Neeraj Singh
2022-03-09 23:03 ` [PATCH v5 2/5] core.fsyncmethod: add writeout-only mode Neeraj Singh via GitGitGadget
2022-03-09 23:48 ` Junio C Hamano
2022-03-09 23:03 ` [PATCH v5 3/5] core.fsync: introduce granular fsync control Neeraj Singh via GitGitGadget
2022-03-10 0:21 ` Junio C Hamano
2022-03-10 2:53 ` Neeraj Singh
2022-03-10 7:19 ` Junio C Hamano
2022-03-10 18:38 ` Neeraj Singh
2022-03-10 18:44 ` Junio C Hamano
2022-03-10 19:57 ` Junio C Hamano
2022-03-10 20:25 ` Neeraj Singh
2022-03-10 21:17 ` Junio C Hamano
2022-03-10 13:11 ` Johannes Schindelin
2022-03-10 17:18 ` Junio C Hamano
2022-03-09 23:03 ` [PATCH v5 4/5] core.fsync: new option to harden the index Neeraj Singh via GitGitGadget
2022-03-09 23:03 ` [PATCH v5 5/5] core.fsync: documentation and user-friendly aggregate options Neeraj Singh via GitGitGadget
2022-03-10 9:53 ` Future-proofed syncing of refs Patrick Steinhardt
2022-03-10 9:53 ` [PATCH 6/8] core.fsync: add `fsync_component()` wrapper which doesn't die Patrick Steinhardt
2022-03-10 17:34 ` Junio C Hamano
2022-03-10 18:40 ` Neeraj Singh
2022-03-10 9:53 ` [PATCH 7/8] core.fsync: new option to harden loose references Patrick Steinhardt
2022-03-10 18:25 ` Junio C Hamano
2022-03-10 19:03 ` Neeraj Singh
2022-03-10 22:54 ` Neeraj Singh
2022-03-11 6:40 ` Junio C Hamano
2022-03-11 9:15 ` Patrick Steinhardt
2022-03-11 9:36 ` Ævar Arnfjörð Bjarmason
2022-03-10 9:53 ` [PATCH 8/8] core.fsync: new option to harden packed references Patrick Steinhardt
2022-03-10 18:28 ` Junio C Hamano
2022-03-11 9:10 ` Patrick Steinhardt
2022-03-10 22:43 ` [PATCH v6 0/6] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget
2022-03-10 22:43 ` [PATCH v6 1/6] wrapper: make inclusion of Windows csprng header tightly scoped Neeraj Singh via GitGitGadget
2022-03-10 22:43 ` [PATCH v6 2/6] core.fsyncmethod: add writeout-only mode Neeraj Singh via GitGitGadget
2022-03-10 22:43 ` [PATCH v6 3/6] core.fsync: introduce granular fsync control infrastructure Neeraj Singh via GitGitGadget
2022-03-10 22:43 ` [PATCH v6 4/6] core.fsync: add configuration parsing Neeraj Singh via GitGitGadget
2022-03-28 11:06 ` Jiang Xin
2022-03-28 19:45 ` Neeraj Singh
2022-03-10 22:43 ` [PATCH v6 5/6] core.fsync: new option to harden the index Neeraj Singh via GitGitGadget
2022-03-10 22:43 ` [PATCH v6 6/6] core.fsync: documentation and user-friendly aggregate options Neeraj Singh via GitGitGadget
2022-03-15 19:12 ` [PATCH v7] " Neeraj Singh
2022-03-15 19:32 ` Junio C Hamano
2022-03-15 19:56 ` Neeraj Singh
2022-03-23 14:20 ` do we have too much fsync() configuration in 'next'? (was: [PATCH v7] core.fsync: documentation and user-friendly aggregate options) Ævar Arnfjörð Bjarmason
2022-03-25 21:24 ` Neeraj Singh
2022-03-26 0:24 ` Ævar Arnfjörð Bjarmason
2022-03-26 1:23 ` do we have too much fsync() configuration in 'next'? Junio C Hamano
2022-03-26 1:25 ` do we have too much fsync() configuration in 'next'? (was: [PATCH v7] core.fsync: documentation and user-friendly aggregate options) Neeraj Singh
2022-03-26 15:31 ` Ævar Arnfjörð Bjarmason
2022-03-27 5:27 ` Neeraj Singh
2022-03-27 12:43 ` Ævar Arnfjörð Bjarmason
2022-03-28 10:56 ` Patrick Steinhardt
2022-03-28 11:25 ` Ævar Arnfjörð Bjarmason
2022-03-28 19:56 ` Neeraj Singh
2022-03-30 16:59 ` Neeraj Singh
2022-03-10 23:34 ` [PATCH v6 0/6] A design for future-proofing fsync() configuration Junio C Hamano
2022-03-11 0:03 ` Neeraj Singh
2022-03-11 18:50 ` Neeraj Singh
2022-03-13 23:50 ` Junio C Hamano
2022-03-11 9:58 ` [PATCH v2] core.fsync: new option to harden references Patrick Steinhardt
2022-03-25 6:11 ` SZEDER Gábor
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).