* [PATCH 0/2] A design for future-proofing fsync() configuration
@ 2021-12-04 3:28 Neeraj K. Singh via GitGitGadget
2021-12-04 3:28 ` [PATCH 1/2] fsync: add writeout-only mode for fsyncing repo data Neeraj Singh via GitGitGadget
` (2 more replies)
0 siblings, 3 replies; 122+ messages in thread
From: Neeraj K. Singh via GitGitGadget @ 2021-12-04 3:28 UTC (permalink / raw)
To: git; +Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps,
Neeraj K. Singh
This is an implementation of an extensible configuration mechanism for
fsyncing persistent components of a repo.
The main goals are to separate the "what" to sync from the "how". There are
now two settings: core.fsync - Control the 'what', including the index.
core.fsyncMethod - Control the 'how'. Currently we support writeout-only and
full fsync.
Syncing of refs can be layered on top of core.fsync. And batch mode will be
layered on core.fsyncMethod.
core.fsyncobjectfiles is removed and will issue a deprecation warning if
it's seen.
I'd like to get agreement on this direction before redoing batch mode on top
of this change.
As of this writing, the change isn't tested in detail. I'll be making sure
with strace that each component has the desired effect.
Please see [1], [2], and [3] for discussions that led to this series.
[1] https://lore.kernel.org/git/211110.86r1bogg27.gmgdl@evledraar.gmail.com/
[2]
https://lore.kernel.org/git/dd65718814011eb93ccc4428f9882e0f025224a6.1636029491.git.ps@pks.im/
[3]
https://lore.kernel.org/git/pull.1076.git.git.1629856292.gitgitgadget@gmail.com/
Neeraj Singh (2):
fsync: add writeout-only mode for fsyncing repo data
core.fsync: introduce granular fsync control
Documentation/config/core.txt | 35 +++++++++---
builtin/fast-import.c | 2 +-
builtin/index-pack.c | 4 +-
builtin/pack-objects.c | 8 ++-
bulk-checkin.c | 5 +-
cache.h | 48 +++++++++++++++-
commit-graph.c | 3 +-
compat/mingw.h | 3 +
compat/win32/flush.c | 28 +++++++++
config.c | 89 ++++++++++++++++++++++++++++-
config.mak.uname | 3 +
configure.ac | 8 +++
contrib/buildsystems/CMakeLists.txt | 3 +-
csum-file.c | 5 +-
csum-file.h | 2 +-
environment.c | 3 +-
git-compat-util.h | 24 ++++++++
midx.c | 3 +-
object-file.c | 3 +-
pack-bitmap-write.c | 3 +-
pack-write.c | 12 ++--
read-cache.c | 19 ++++--
wrapper.c | 56 ++++++++++++++++++
write-or-die.c | 10 ++--
24 files changed, 337 insertions(+), 42 deletions(-)
create mode 100644 compat/win32/flush.c
base-commit: abe6bb3905392d5eb6b01fa6e54d7e784e0522aa
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1093%2Fneerajsi-msft%2Fns%2Fcore-fsync-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1093/neerajsi-msft/ns/core-fsync-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/1093
--
gitgitgadget
^ permalink raw reply [flat|nested] 122+ messages in thread* [PATCH 1/2] fsync: add writeout-only mode for fsyncing repo data 2021-12-04 3:28 [PATCH 0/2] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget @ 2021-12-04 3:28 ` Neeraj Singh via GitGitGadget 2021-12-06 7:54 ` Neeraj Singh 2021-12-04 3:28 ` [PATCH 2/2] core.fsync: introduce granular fsync control Neeraj Singh via GitGitGadget 2021-12-07 2:46 ` [PATCH v2 0/3] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget 2 siblings, 1 reply; 122+ messages in thread From: Neeraj Singh via GitGitGadget @ 2021-12-04 3:28 UTC (permalink / raw) To: git Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps, Neeraj K. Singh, Neeraj Singh From: Neeraj Singh <neerajsi@microsoft.com> The new writeout-only mode attempts to tell the operating system to flush its in-memory page cache to the storage hardware without issuing a CACHE_FLUSH command to the storage controller. Writeout-only fsync is significantly faster than a vanilla fsync on common hardware, since data is written to a disk-side cache rather than all the way to a durable medium. Later changes in this patch series will take advantage of this primitive to implement batching of hardware flushes. When git_fsync is called with FSYNC_WRITEOUT_ONLY, it may fail and the caller is expected to do an ordinary fsync as needed. On Apple platforms, the fsync system call does not issue a CACHE_FLUSH directive to the storage controller. This change updates fsync to do fcntl(F_FULLFSYNC) to make fsync actually durable. We maintain parity with existing behavior on Apple platforms by setting the default value of the new core.fsyncmethod option. Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> --- Documentation/config/core.txt | 9 +++++ cache.h | 7 ++++ compat/mingw.h | 3 ++ compat/win32/flush.c | 28 +++++++++++++++ config.c | 12 +++++++ config.mak.uname | 3 ++ configure.ac | 8 +++++ contrib/buildsystems/CMakeLists.txt | 3 +- environment.c | 2 +- git-compat-util.h | 24 +++++++++++++ wrapper.c | 56 +++++++++++++++++++++++++++++ write-or-die.c | 10 +++--- 12 files changed, 159 insertions(+), 6 deletions(-) create mode 100644 compat/win32/flush.c diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt index c04f62a54a1..c91eccea598 100644 --- a/Documentation/config/core.txt +++ b/Documentation/config/core.txt @@ -547,6 +547,15 @@ core.whitespace:: is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent` errors. The default tab width is 8. Allowed values are 1 to 63. +core.fsyncMethod:: + A value indicating the strategy Git will use to harden repository data + using fsync and related primitives. ++ +* `fsync` uses the fsync() system call or platform equivalents. +* `writeout-only` issues requests to send the writes to the storage + hardware, but does not send any FLUSH CACHE request. If the operating system + does not support the required interfaces, this falls back to fsync(). + core.fsyncObjectFiles:: This boolean will enable 'fsync()' when writing object files. + diff --git a/cache.h b/cache.h index eba12487b99..9cd60d94952 100644 --- a/cache.h +++ b/cache.h @@ -986,6 +986,13 @@ extern int read_replace_refs; extern char *git_replace_ref_base; extern int fsync_object_files; + +enum fsync_method { + FSYNC_METHOD_FSYNC, + FSYNC_METHOD_WRITEOUT_ONLY +}; + +extern enum fsync_method fsync_method; extern int core_preload_index; extern int precomposed_unicode; extern int protect_hfs; diff --git a/compat/mingw.h b/compat/mingw.h index c9a52ad64a6..6074a3d3ced 100644 --- a/compat/mingw.h +++ b/compat/mingw.h @@ -329,6 +329,9 @@ int mingw_getpagesize(void); #define getpagesize mingw_getpagesize #endif +int win32_fsync_no_flush(int fd); +#define fsync_no_flush win32_fsync_no_flush + struct rlimit { unsigned int rlim_cur; }; diff --git a/compat/win32/flush.c b/compat/win32/flush.c new file mode 100644 index 00000000000..75324c24ee7 --- /dev/null +++ b/compat/win32/flush.c @@ -0,0 +1,28 @@ +#include "../../git-compat-util.h" +#include <winternl.h> +#include "lazyload.h" + +int win32_fsync_no_flush(int fd) +{ + IO_STATUS_BLOCK io_status; + +#define FLUSH_FLAGS_FILE_DATA_ONLY 1 + + DECLARE_PROC_ADDR(ntdll.dll, NTSTATUS, NtFlushBuffersFileEx, + HANDLE FileHandle, ULONG Flags, PVOID Parameters, ULONG ParameterSize, + PIO_STATUS_BLOCK IoStatusBlock); + + if (!INIT_PROC_ADDR(NtFlushBuffersFileEx)) { + errno = ENOSYS; + return -1; + } + + memset(&io_status, 0, sizeof(io_status)); + if (NtFlushBuffersFileEx((HANDLE)_get_osfhandle(fd), FLUSH_FLAGS_FILE_DATA_ONLY, + NULL, 0, &io_status)) { + errno = EINVAL; + return -1; + } + + return 0; +} diff --git a/config.c b/config.c index c5873f3a706..c3410b8a868 100644 --- a/config.c +++ b/config.c @@ -1490,6 +1490,18 @@ static int git_default_core_config(const char *var, const char *value, void *cb) return 0; } + if (!strcmp(var, "core.fsyncmethod")) { + if (!value) + return config_error_nonbool(var); + if (!strcmp(value, "fsync")) + fsync_method = FSYNC_METHOD_FSYNC; + else if (!strcmp(value, "writeout-only")) + fsync_method = FSYNC_METHOD_WRITEOUT_ONLY; + else + warning(_("unknown %s value '%s'"), var, value); + + } + if (!strcmp(var, "core.fsyncobjectfiles")) { fsync_object_files = git_config_bool(var, value); return 0; diff --git a/config.mak.uname b/config.mak.uname index d0701f9beb0..774a09622d2 100644 --- a/config.mak.uname +++ b/config.mak.uname @@ -57,6 +57,7 @@ ifeq ($(uname_S),Linux) HAVE_CLOCK_MONOTONIC = YesPlease # -lrt is needed for clock_gettime on glibc <= 2.16 NEEDS_LIBRT = YesPlease + HAVE_SYNC_FILE_RANGE = YesPlease HAVE_GETDELIM = YesPlease FREAD_READS_DIRECTORIES = UnfortunatelyYes BASIC_CFLAGS += -DHAVE_SYSINFO @@ -453,6 +454,7 @@ endif CFLAGS = BASIC_CFLAGS = -nologo -I. -Icompat/vcbuild/include -DWIN32 -D_CONSOLE -DHAVE_STRING_H -D_CRT_SECURE_NO_WARNINGS -D_CRT_NONSTDC_NO_DEPRECATE COMPAT_OBJS = compat/msvc.o compat/winansi.o \ + compat/win32/flush.o \ compat/win32/path-utils.o \ compat/win32/pthread.o compat/win32/syslog.o \ compat/win32/trace2_win32_process_info.o \ @@ -628,6 +630,7 @@ ifeq ($(uname_S),MINGW) COMPAT_CFLAGS += -DSTRIP_EXTENSION=\".exe\" COMPAT_OBJS += compat/mingw.o compat/winansi.o \ compat/win32/trace2_win32_process_info.o \ + compat/win32/flush.o \ compat/win32/path-utils.o \ compat/win32/pthread.o compat/win32/syslog.o \ compat/win32/dirent.o diff --git a/configure.ac b/configure.ac index 5ee25ec95c8..6bd6bef1c44 100644 --- a/configure.ac +++ b/configure.ac @@ -1082,6 +1082,14 @@ AC_COMPILE_IFELSE([CLOCK_MONOTONIC_SRC], [AC_MSG_RESULT([no]) HAVE_CLOCK_MONOTONIC=]) GIT_CONF_SUBST([HAVE_CLOCK_MONOTONIC]) + +# +# Define HAVE_SYNC_FILE_RANGE=YesPlease if sync_file_range is available. +GIT_CHECK_FUNC(sync_file_range, + [HAVE_SYNC_FILE_RANGE=YesPlease], + [HAVE_SYNC_FILE_RANGE]) +GIT_CONF_SUBST([HAVE_SYNC_FILE_RANGE]) + # # Define NO_SETITIMER if you don't have setitimer. GIT_CHECK_FUNC(setitimer, diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt index 86b46114464..6d7bc16d054 100644 --- a/contrib/buildsystems/CMakeLists.txt +++ b/contrib/buildsystems/CMakeLists.txt @@ -261,7 +261,8 @@ if(CMAKE_SYSTEM_NAME STREQUAL "Windows") NOGDI OBJECT_CREATION_MODE=1 __USE_MINGW_ANSI_STDIO=0 USE_NED_ALLOCATOR OVERRIDE_STRDUP MMAP_PREVENTS_DELETE USE_WIN32_MMAP UNICODE _UNICODE HAVE_WPGMPTR ENSURE_MSYSTEM_IS_SET) - list(APPEND compat_SOURCES compat/mingw.c compat/winansi.c compat/win32/path-utils.c + list(APPEND compat_SOURCES compat/mingw.c compat/winansi.c + compat/win32/flush.c compat/win32/path-utils.c compat/win32/pthread.c compat/win32mmap.c compat/win32/syslog.c compat/win32/trace2_win32_process_info.c compat/win32/dirent.c compat/nedmalloc/nedmalloc.c compat/strdup.c) diff --git a/environment.c b/environment.c index 9da7f3c1a19..f9140e842cf 100644 --- a/environment.c +++ b/environment.c @@ -41,7 +41,7 @@ const char *git_attributes_file; const char *git_hooks_path; int zlib_compression_level = Z_BEST_SPEED; int pack_compression_level = Z_DEFAULT_COMPRESSION; -int fsync_object_files; +enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT; size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE; size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT; size_t delta_base_cache_limit = 96 * 1024 * 1024; diff --git a/git-compat-util.h b/git-compat-util.h index c6bd2a84e55..cb9abd7a08c 100644 --- a/git-compat-util.h +++ b/git-compat-util.h @@ -1239,6 +1239,30 @@ __attribute__((format (printf, 1, 2))) NORETURN void BUG(const char *fmt, ...); #endif +#ifdef __APPLE__ +#define FSYNC_METHOD_DEFAULT FSYNC_METHOD_WRITEOUT_ONLY +#else +#define FSYNC_METHOD_DEFAULT FSYNC_METHOD_FSYNC +#endif + +enum fsync_action { + FSYNC_WRITEOUT_ONLY, + FSYNC_HARDWARE_FLUSH +}; + +/* + * Issues an fsync against the specified file according to the specified mode. + * + * FSYNC_WRITEOUT_ONLY attempts to use interfaces available on some operating + * systems to flush the OS cache without issuing a flush command to the storage + * controller. If those interfaces are unavailable, the function fails with + * ENOSYS. + * + * FSYNC_HARDWARE_FLUSH does an OS writeout and hardware flush to ensure that + * changes are durable. It is not expected to fail. + */ +int git_fsync(int fd, enum fsync_action action); + /* * Preserves errno, prints a message, but gives no warning for ENOENT. * Returns 0 on success, which includes trying to unlink an object that does diff --git a/wrapper.c b/wrapper.c index 36e12119d76..1c5f2c87791 100644 --- a/wrapper.c +++ b/wrapper.c @@ -546,6 +546,62 @@ int xmkstemp_mode(char *filename_template, int mode) return fd; } +int git_fsync(int fd, enum fsync_action action) +{ + switch (action) { + case FSYNC_WRITEOUT_ONLY: + +#ifdef __APPLE__ + /* + * on macOS, fsync just causes filesystem cache writeback but does not + * flush hardware caches. + */ + return fsync(fd); +#endif + +#ifdef HAVE_SYNC_FILE_RANGE + /* + * On linux 2.6.17 and above, sync_file_range is the way to issue + * a writeback without a hardware flush. An offset of 0 and size of 0 + * indicates writeout of the entire file and the wait flags ensure that all + * dirty data is written to the disk (potentially in a disk-side cache) + * before we continue. + */ + + return sync_file_range(fd, 0, 0, SYNC_FILE_RANGE_WAIT_BEFORE | + SYNC_FILE_RANGE_WRITE | + SYNC_FILE_RANGE_WAIT_AFTER); +#endif + +#ifdef fsync_no_flush + return fsync_no_flush(fd); +#endif + + errno = ENOSYS; + return -1; + + case FSYNC_HARDWARE_FLUSH: + /* + * On some platforms fsync may return EINTR. Try again in this + * case, since callers asking for a hardware flush may die if + * this function returns an error. + */ + for (;;) { + int err; +#ifdef __APPLE__ + err = fcntl(fd, F_FULLFSYNC); +#else + err = fsync(fd); +#endif + if (err >= 0 || errno != EINTR) + return err; + } + + default: + BUG("unexpected git_fsync(%d) call", action); + } +} + static int warn_if_unremovable(const char *op, const char *file, int rc) { int err; diff --git a/write-or-die.c b/write-or-die.c index 0b1ec8190b6..0702acdd5e8 100644 --- a/write-or-die.c +++ b/write-or-die.c @@ -57,10 +57,12 @@ void fprintf_or_die(FILE *f, const char *fmt, ...) void fsync_or_die(int fd, const char *msg) { - while (fsync(fd) < 0) { - if (errno != EINTR) - die_errno("fsync error on '%s'", msg); - } + if (fsync_method == FSYNC_METHOD_WRITEOUT_ONLY && + git_fsync(fd, FSYNC_WRITEOUT_ONLY) >= 0) + return; + + if (git_fsync(fd, FSYNC_HARDWARE_FLUSH) < 0) + die_errno("fsync error on '%s'", msg); } void write_or_die(int fd, const void *buf, size_t count) -- gitgitgadget ^ permalink raw reply related [flat|nested] 122+ messages in thread
* Re: [PATCH 1/2] fsync: add writeout-only mode for fsyncing repo data 2021-12-04 3:28 ` [PATCH 1/2] fsync: add writeout-only mode for fsyncing repo data Neeraj Singh via GitGitGadget @ 2021-12-06 7:54 ` Neeraj Singh 0 siblings, 0 replies; 122+ messages in thread From: Neeraj Singh @ 2021-12-06 7:54 UTC (permalink / raw) To: Neeraj Singh via GitGitGadget Cc: git, rsbecker, bagasdotme, newren, avarab, ps, Neeraj K. Singh Context for reviewers: The mechanism part of this patch is equivalent to [1] and [2]. With the following changes: * The configuration code is a bit different and we now expose a `writeout-only` mode to the user. This mode is the default on macOS to prevent a change in end-user behavior. * git_fsync now contains the EINTR retry loop internally. [1] https://lore.kernel.org/git/e1747ce00af7ab3170a69955b07d995d5321d6f3.1637020263.git.gitgitgadget@gmail.com/ [2] https://lore.kernel.org/git/546ad9c82e8e0c2eb4683f9f360d8f30e2136020.1630108177.git.gitgitgadget@gmail.com/ ^ permalink raw reply [flat|nested] 122+ messages in thread
* [PATCH 2/2] core.fsync: introduce granular fsync control 2021-12-04 3:28 [PATCH 0/2] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget 2021-12-04 3:28 ` [PATCH 1/2] fsync: add writeout-only mode for fsyncing repo data Neeraj Singh via GitGitGadget @ 2021-12-04 3:28 ` Neeraj Singh via GitGitGadget 2021-12-07 2:46 ` [PATCH v2 0/3] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget 2 siblings, 0 replies; 122+ messages in thread From: Neeraj Singh via GitGitGadget @ 2021-12-04 3:28 UTC (permalink / raw) To: git Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps, Neeraj K. Singh, Neeraj Singh From: Neeraj Singh <neerajsi@microsoft.com> This commit introduces the `core.fsync` configuration knob which can be used to control how components of the repository are made durable on disk. This setting allows future extensibility of components that could be synced in two ways: * We issue a warning rather than an error for unrecognized components, so new configs can be used with old Git versions. * We support negation, so users can choose one of the default aggregate options and then remove components that they don't want. This also support the common request of doing absolutely no fysncing with the `core.fsync=none` value, which is expected to make the test suite faster. This commit introduces the new ability for the user to harden the index, which is a requirement for being able to actually find a file that has been added to the repo and then deleted from the working tree. Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> --- Documentation/config/core.txt | 28 +++++++++---- builtin/fast-import.c | 2 +- builtin/index-pack.c | 4 +- builtin/pack-objects.c | 8 ++-- bulk-checkin.c | 5 ++- cache.h | 41 ++++++++++++++++++- commit-graph.c | 3 +- config.c | 77 ++++++++++++++++++++++++++++++++++- csum-file.c | 5 ++- csum-file.h | 2 +- environment.c | 1 + midx.c | 3 +- object-file.c | 3 +- pack-bitmap-write.c | 3 +- pack-write.c | 12 +++--- read-cache.c | 19 ++++++--- 16 files changed, 179 insertions(+), 37 deletions(-) diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt index c91eccea598..d502f8a1bf5 100644 --- a/Documentation/config/core.txt +++ b/Documentation/config/core.txt @@ -547,6 +547,26 @@ core.whitespace:: is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent` errors. The default tab width is 8. Allowed values are 1 to 63. +core.fsync:: + A comma-separated list of parts of the repository which should be + hardened via the core.fsyncMethod when created or modified. You can + disable hardening of any component by prefixing it with a '-'. Later + items take precedence over earlier ones in the list. For example, + `core.fsync=all,-index` means "harden everything except the index". + Items that are not hardened may be lost in the event of an unclean + system shutdown. ++ +* `none` disables fsync completely. This must be specified alone. +* `loose-object` hardens objects added to the repo in loose-object form. +* `pack` hardens objects added to the repo in packfile form. +* `pack-metadata` hardens packfile bitmaps and indexes. +* `commit-graph` hardens the commit graph file. +* `index` hardens the index when it is modified. +* `objects` is an aggregate option that includes `loose-objects`, `pack`, + `pack-metadata`, and `commit-graph`. +* `default` is an aggregate option that is equivalent to `objects,-loose-object` +* `all` is an aggregate option that syncs all individual components above. + core.fsyncMethod:: A value indicating the strategy Git will use to harden repository data using fsync and related primitives. @@ -556,14 +576,6 @@ core.fsyncMethod:: hardware, but does not send any FLUSH CACHE request. If the operating system does not support the required interfaces, this falls back to fsync(). -core.fsyncObjectFiles:: - This boolean will enable 'fsync()' when writing object files. -+ -This is a total waste of time and effort on a filesystem that orders -data writes properly, but can be useful for filesystems that do not use -journalling (traditional UNIX filesystems) or that only journal metadata -and not file contents (OS X's HFS+, or Linux ext3 with "data=writeback"). - core.preloadIndex:: Enable parallel index preload for operations like 'git diff' + diff --git a/builtin/fast-import.c b/builtin/fast-import.c index 20406f67754..0ae17d63618 100644 --- a/builtin/fast-import.c +++ b/builtin/fast-import.c @@ -856,7 +856,7 @@ static void end_packfile(void) struct tag *t; close_pack_windows(pack_data); - finalize_hashfile(pack_file, cur_pack_oid.hash, 0); + finalize_hashfile(pack_file, cur_pack_oid.hash, REPO_COMPONENT_PACK, 0); fixup_pack_header_footer(pack_data->pack_fd, pack_data->hash, pack_data->pack_name, object_count, cur_pack_oid.hash, pack_size); diff --git a/builtin/index-pack.c b/builtin/index-pack.c index c23d01de7dc..9157a955de7 100644 --- a/builtin/index-pack.c +++ b/builtin/index-pack.c @@ -1286,7 +1286,7 @@ static void conclude_pack(int fix_thin_pack, const char *curr_pack, unsigned cha nr_objects - nr_objects_initial); stop_progress_msg(&progress, msg.buf); strbuf_release(&msg); - finalize_hashfile(f, tail_hash, 0); + finalize_hashfile(f, tail_hash, REPO_COMPONENT_PACK, 0); hashcpy(read_hash, pack_hash); fixup_pack_header_footer(output_fd, pack_hash, curr_pack, nr_objects, @@ -1508,7 +1508,7 @@ static void final(const char *final_pack_name, const char *curr_pack_name, if (!from_stdin) { close(input_fd); } else { - fsync_or_die(output_fd, curr_pack_name); + fsync_component_or_die(REPO_COMPONENT_PACK, output_fd, curr_pack_name); err = close(output_fd); if (err) die_errno(_("error while closing pack file")); diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 857be7826f3..48c2f9f3847 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -1204,11 +1204,13 @@ static void write_pack_file(void) * If so, rewrite it like in fast-import */ if (pack_to_stdout) { - finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_CLOSE); + finalize_hashfile(f, hash, REPO_COMPONENT_NONE, + CSUM_HASH_IN_STREAM | CSUM_CLOSE); } else if (nr_written == nr_remaining) { - finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); + finalize_hashfile(f, hash, REPO_COMPONENT_PACK, + CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); } else { - int fd = finalize_hashfile(f, hash, 0); + int fd = finalize_hashfile(f, hash, REPO_COMPONENT_PACK, 0); fixup_pack_header_footer(fd, hash, pack_tmp_name, nr_written, hash, offset); close(fd); diff --git a/bulk-checkin.c b/bulk-checkin.c index 8785b2ac806..b9f3d315334 100644 --- a/bulk-checkin.c +++ b/bulk-checkin.c @@ -53,9 +53,10 @@ static void finish_bulk_checkin(struct bulk_checkin_state *state) unlink(state->pack_tmp_name); goto clear_exit; } else if (state->nr_written == 1) { - finalize_hashfile(state->f, hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); + finalize_hashfile(state->f, hash, REPO_COMPONENT_PACK, + CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); } else { - int fd = finalize_hashfile(state->f, hash, 0); + int fd = finalize_hashfile(state->f, hash, REPO_COMPONENT_PACK, 0); fixup_pack_header_footer(fd, hash, state->pack_tmp_name, state->nr_written, hash, state->offset); diff --git a/cache.h b/cache.h index 9cd60d94952..b2966352440 100644 --- a/cache.h +++ b/cache.h @@ -985,7 +985,40 @@ void reset_shared_repository(void); extern int read_replace_refs; extern char *git_replace_ref_base; -extern int fsync_object_files; +/* + * These values are used to help identify parts of a repository to fsync. + * REPO_COMPONENT_NONE identifies data that will not be a persistent part of the + * repository and so shouldn't be fsynced. + */ +enum repo_component { + REPO_COMPONENT_NONE = 0, + REPO_COMPONENT_LOOSE_OBJECT = 1 << 0, + REPO_COMPONENT_PACK = 1 << 1, + REPO_COMPONENT_PACK_METADATA = 1 << 2, + REPO_COMPONENT_COMMIT_GRAPH = 1 << 3, + REPO_COMPONENT_INDEX = 1 << 4, +}; + +#define FSYNC_COMPONENTS_DEFAULT (REPO_COMPONENT_PACK | \ + REPO_COMPONENT_PACK_METADATA | \ + REPO_COMPONENT_COMMIT_GRAPH) + +#define FSYNC_COMPONENTS_OBJECTS (REPO_COMPONENT_LOOSE_OBJECT | \ + REPO_COMPONENT_PACK | \ + REPO_COMPONENT_PACK_METADATA | \ + REPO_COMPONENT_COMMIT_GRAPH) + +#define FSYNC_COMPONENTS_ALL (REPO_COMPONENT_LOOSE_OBJECT | \ + REPO_COMPONENT_PACK | \ + REPO_COMPONENT_PACK_METADATA | \ + REPO_COMPONENT_COMMIT_GRAPH | \ + REPO_COMPONENT_INDEX) + + +/* + * A bitmask indicating which components of the repo should be fsynced. + */ +extern enum repo_component fsync_components; enum fsync_method { FSYNC_METHOD_FSYNC, @@ -1747,6 +1780,12 @@ int copy_file_with_time(const char *dst, const char *src, int mode); void write_or_die(int fd, const void *buf, size_t count); void fsync_or_die(int fd, const char *); +inline void fsync_component_or_die(enum repo_component component, int fd, const char *msg) +{ + if (fsync_components & component) + fsync_or_die(fd, msg); +} + ssize_t read_in_full(int fd, void *buf, size_t count); ssize_t write_in_full(int fd, const void *buf, size_t count); ssize_t pread_in_full(int fd, void *buf, size_t count, off_t offset); diff --git a/commit-graph.c b/commit-graph.c index 2706683acfe..4bed4175ab2 100644 --- a/commit-graph.c +++ b/commit-graph.c @@ -1939,7 +1939,8 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx) } close_commit_graph(ctx->r->objects); - finalize_hashfile(f, file_hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC); + finalize_hashfile(f, file_hash, REPO_COMPONENT_COMMIT_GRAPH, + CSUM_HASH_IN_STREAM | CSUM_FSYNC); free_chunkfile(cf); if (ctx->split) { diff --git a/config.c b/config.c index c3410b8a868..6c8b102ed7a 100644 --- a/config.c +++ b/config.c @@ -1213,6 +1213,74 @@ static int git_parse_maybe_bool_text(const char *value) return -1; } +static const struct fsync_component_entry { + const char *name; + enum repo_component component_bits; +} fsync_component_table[] = { + { "loose-object", REPO_COMPONENT_LOOSE_OBJECT }, + { "pack", REPO_COMPONENT_PACK }, + { "pack-metadata", REPO_COMPONENT_PACK_METADATA }, + { "commit-graph", REPO_COMPONENT_COMMIT_GRAPH }, + { "index", REPO_COMPONENT_INDEX }, + { "objects", FSYNC_COMPONENTS_OBJECTS }, + { "default", FSYNC_COMPONENTS_DEFAULT }, + { "all", FSYNC_COMPONENTS_ALL }, +}; + +static enum repo_component parse_fsync_components(const char *var, const char *string) +{ + enum repo_component output = 0; + + if (!strcmp(string, "none")) + return output; + + while (string) { + int i; + size_t len; + const char *ep; + int negated = 0; + int found = 0; + + string = string + strspn(string, ", \t\n\r"); + ep = strchrnul(string, ','); + len = ep - string; + + if (*string == '-') { + negated = 1; + string++; + len--; + if (!len) + warning(_("invalid value for variable %s"), var); + } + + if (!len) + break; + + for (i = 0; i < ARRAY_SIZE(fsync_component_table); ++i) { + const struct fsync_component_entry *entry = &fsync_component_table[i]; + + if (strncmp(entry->name, string, len)) + continue; + + found = 1; + if (negated) + output &= ~entry->component_bits; + else + output |= entry->component_bits; + } + + if (!found) { + char *component = xstrndup(string, len); + warning(_("unknown %s value '%s'"), var, component); + free(component); + } + + string = ep; + } + + return output; +} + int git_parse_maybe_bool(const char *value) { int v = git_parse_maybe_bool_text(value); @@ -1490,6 +1558,13 @@ static int git_default_core_config(const char *var, const char *value, void *cb) return 0; } + if (!strcmp(var, "core.fsync")) { + if (!value) + return config_error_nonbool(var); + fsync_components = parse_fsync_components(var, value); + return 0; + } + if (!strcmp(var, "core.fsyncmethod")) { if (!value) return config_error_nonbool(var); @@ -1503,7 +1578,7 @@ static int git_default_core_config(const char *var, const char *value, void *cb) } if (!strcmp(var, "core.fsyncobjectfiles")) { - fsync_object_files = git_config_bool(var, value); + warning(_("core.fsyncobjectfiles is deprecated; use core.fsync instead")); return 0; } diff --git a/csum-file.c b/csum-file.c index 26e8a6df44e..adc8023d5af 100644 --- a/csum-file.c +++ b/csum-file.c @@ -58,7 +58,8 @@ static void free_hashfile(struct hashfile *f) free(f); } -int finalize_hashfile(struct hashfile *f, unsigned char *result, unsigned int flags) +int finalize_hashfile(struct hashfile *f, unsigned char *result, + enum repo_component component, unsigned int flags) { int fd; @@ -69,7 +70,7 @@ int finalize_hashfile(struct hashfile *f, unsigned char *result, unsigned int fl if (flags & CSUM_HASH_IN_STREAM) flush(f, f->buffer, the_hash_algo->rawsz); if (flags & CSUM_FSYNC) - fsync_or_die(f->fd, f->name); + fsync_component_or_die(component, f->fd, f->name); if (flags & CSUM_CLOSE) { if (close(f->fd)) die_errno("%s: sha1 file error on close", f->name); diff --git a/csum-file.h b/csum-file.h index 291215b34eb..3820b4a0e94 100644 --- a/csum-file.h +++ b/csum-file.h @@ -38,7 +38,7 @@ int hashfile_truncate(struct hashfile *, struct hashfile_checkpoint *); struct hashfile *hashfd(int fd, const char *name); struct hashfile *hashfd_check(const char *name); struct hashfile *hashfd_throughput(int fd, const char *name, struct progress *tp); -int finalize_hashfile(struct hashfile *, unsigned char *, unsigned int); +int finalize_hashfile(struct hashfile *, unsigned char *, enum repo_component, unsigned int); void hashwrite(struct hashfile *, const void *, unsigned int); void hashflush(struct hashfile *f); void crc32_begin(struct hashfile *); diff --git a/environment.c b/environment.c index f9140e842cf..190df463475 100644 --- a/environment.c +++ b/environment.c @@ -42,6 +42,7 @@ const char *git_hooks_path; int zlib_compression_level = Z_BEST_SPEED; int pack_compression_level = Z_DEFAULT_COMPRESSION; enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT; +enum repo_component fsync_components = FSYNC_COMPONENTS_DEFAULT; size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE; size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT; size_t delta_base_cache_limit = 96 * 1024 * 1024; diff --git a/midx.c b/midx.c index 837b46b2af5..6e9510ab0dc 100644 --- a/midx.c +++ b/midx.c @@ -1406,7 +1406,8 @@ static int write_midx_internal(const char *object_dir, write_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs); write_chunkfile(cf, &ctx); - finalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM); + finalize_hashfile(f, midx_hash, REPO_COMPONENT_PACK_METADATA, + CSUM_FSYNC | CSUM_HASH_IN_STREAM); free_chunkfile(cf); if (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP)) diff --git a/object-file.c b/object-file.c index eb972cdccd2..871ea6bcd53 100644 --- a/object-file.c +++ b/object-file.c @@ -1809,8 +1809,7 @@ int hash_object_file(const struct git_hash_algo *algo, const void *buf, /* Finalize a file on disk, and close it. */ static void close_loose_object(int fd) { - if (fsync_object_files) - fsync_or_die(fd, "loose object file"); + fsync_component_or_die(REPO_COMPONENT_LOOSE_OBJECT, fd, "loose object file"); if (close(fd) != 0) die_errno(_("error when closing loose object file")); } diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c index 9c55c1531e1..e82e87af996 100644 --- a/pack-bitmap-write.c +++ b/pack-bitmap-write.c @@ -719,7 +719,8 @@ void bitmap_writer_finish(struct pack_idx_entry **index, if (options & BITMAP_OPT_HASH_CACHE) write_hash_cache(f, index, index_nr); - finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); + finalize_hashfile(f, NULL, REPO_COMPONENT_PACK_METADATA, + CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); if (adjust_shared_perm(tmp_file.buf)) die_errno("unable to make temporary bitmap file readable"); diff --git a/pack-write.c b/pack-write.c index a5846f3a346..d9c37803e98 100644 --- a/pack-write.c +++ b/pack-write.c @@ -159,8 +159,9 @@ const char *write_idx_file(const char *index_name, struct pack_idx_entry **objec } hashwrite(f, sha1, the_hash_algo->rawsz); - finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE | - ((opts->flags & WRITE_IDX_VERIFY) + finalize_hashfile(f, NULL, REPO_COMPONENT_PACK_METADATA, + CSUM_HASH_IN_STREAM | CSUM_CLOSE | + ((opts->flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC)); return index_name; } @@ -281,8 +282,9 @@ const char *write_rev_file_order(const char *rev_name, if (rev_name && adjust_shared_perm(rev_name) < 0) die(_("failed to make %s readable"), rev_name); - finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE | - ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC)); + finalize_hashfile(f, NULL, REPO_COMPONENT_PACK_METADATA, + CSUM_HASH_IN_STREAM | CSUM_CLOSE | + ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC)); return rev_name; } @@ -390,7 +392,7 @@ void fixup_pack_header_footer(int pack_fd, the_hash_algo->final_fn(partial_pack_hash, &old_hash_ctx); the_hash_algo->final_fn(new_pack_hash, &new_hash_ctx); write_or_die(pack_fd, new_pack_hash, the_hash_algo->rawsz); - fsync_or_die(pack_fd, pack_name); + fsync_component_or_die(REPO_COMPONENT_PACK, pack_fd, pack_name); } char *index_pack_lockfile(int ip_out, int *is_well_formed) diff --git a/read-cache.c b/read-cache.c index f3986596623..883d0c0019a 100644 --- a/read-cache.c +++ b/read-cache.c @@ -2816,7 +2816,7 @@ static int record_ieot(void) * rely on it. */ static int do_write_index(struct index_state *istate, struct tempfile *tempfile, - int strip_extensions) + int strip_extensions, unsigned flags) { uint64_t start = getnanotime(); struct hashfile *f; @@ -2830,6 +2830,7 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile, struct strbuf previous_name_buf = STRBUF_INIT, *previous_name; int drop_cache_tree = istate->drop_cache_tree; off_t offset; + int csum_fsync_flag; int ieot_entries = 1; struct index_entry_offset_table *ieot = NULL; int nr, nr_threads; @@ -3060,7 +3061,13 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile, return -1; } - finalize_hashfile(f, istate->oid.hash, CSUM_HASH_IN_STREAM); + csum_fsync_flag = 0; + if (!alternate_index_output && (flags & COMMIT_LOCK)) + csum_fsync_flag = CSUM_FSYNC; + + finalize_hashfile(f, istate->oid.hash, REPO_COMPONENT_INDEX, + CSUM_HASH_IN_STREAM | csum_fsync_flag); + if (close_tempfile_gently(tempfile)) { error(_("could not close '%s'"), get_tempfile_path(tempfile)); return -1; @@ -3115,7 +3122,7 @@ static int do_write_locked_index(struct index_state *istate, struct lock_file *l */ trace2_region_enter_printf("index", "do_write_index", the_repository, "%s", get_lock_file_path(lock)); - ret = do_write_index(istate, lock->tempfile, 0); + ret = do_write_index(istate, lock->tempfile, 0, flags); trace2_region_leave_printf("index", "do_write_index", the_repository, "%s", get_lock_file_path(lock)); @@ -3209,7 +3216,7 @@ static int clean_shared_index_files(const char *current_hex) } static int write_shared_index(struct index_state *istate, - struct tempfile **temp) + struct tempfile **temp, unsigned flags) { struct split_index *si = istate->split_index; int ret, was_full = !istate->sparse_index; @@ -3219,7 +3226,7 @@ static int write_shared_index(struct index_state *istate, trace2_region_enter_printf("index", "shared/do_write_index", the_repository, "%s", get_tempfile_path(*temp)); - ret = do_write_index(si->base, *temp, 1); + ret = do_write_index(si->base, *temp, 1, flags); trace2_region_leave_printf("index", "shared/do_write_index", the_repository, "%s", get_tempfile_path(*temp)); @@ -3328,7 +3335,7 @@ int write_locked_index(struct index_state *istate, struct lock_file *lock, ret = do_write_locked_index(istate, lock, flags); goto out; } - ret = write_shared_index(istate, &temp); + ret = write_shared_index(istate, &temp, flags); saved_errno = errno; if (is_tempfile_active(temp)) -- gitgitgadget ^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v2 0/3] A design for future-proofing fsync() configuration 2021-12-04 3:28 [PATCH 0/2] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget 2021-12-04 3:28 ` [PATCH 1/2] fsync: add writeout-only mode for fsyncing repo data Neeraj Singh via GitGitGadget 2021-12-04 3:28 ` [PATCH 2/2] core.fsync: introduce granular fsync control Neeraj Singh via GitGitGadget @ 2021-12-07 2:46 ` Neeraj K. Singh via GitGitGadget 2021-12-07 2:46 ` [PATCH v2 1/3] core.fsyncmethod: add writeout-only mode Neeraj Singh via GitGitGadget ` (4 more replies) 2 siblings, 5 replies; 122+ messages in thread From: Neeraj K. Singh via GitGitGadget @ 2021-12-07 2:46 UTC (permalink / raw) To: git; +Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps, Neeraj K. Singh This is an implementation of an extensible configuration mechanism for fsyncing persistent components of a repo. The main goals are to separate the "what" to sync from the "how". There are now two settings: core.fsync - Control the 'what', including the index. core.fsyncMethod - Control the 'how'. Currently we support writeout-only and full fsync. Syncing of refs can be layered on top of core.fsync. And batch mode will be layered on core.fsyncMethod. core.fsyncobjectfiles is removed and will issue a deprecation warning if it's seen. I'd like to get agreement on this direction before submitting batch mode to the list. The batch mode series is available to view at https://github.com/neerajsi-msft/git/pull/1. Please see [1], [2], and [3] for discussions that led to this series. V2 changes: * Updated the documentation for core.fsyncmethod to be less certain. writeout-only probably does not do the right thing on Linux. * Split out the core.fsync=index change into its own commit. * Rename REPO_COMPONENT to FSYNC_COMPONENT. This is really specific to fsyncing, so the name should reflect that. * Re-add missing Makefile change for SYNC_FILE_RANGE. * Tested writeout-only mode, index syncing, and general config settings. [1] https://lore.kernel.org/git/211110.86r1bogg27.gmgdl@evledraar.gmail.com/ [2] https://lore.kernel.org/git/dd65718814011eb93ccc4428f9882e0f025224a6.1636029491.git.ps@pks.im/ [3] https://lore.kernel.org/git/pull.1076.git.git.1629856292.gitgitgadget@gmail.com/ Neeraj Singh (3): core.fsyncmethod: add writeout-only mode core.fsync: introduce granular fsync control core.fsync: new option to harden the index Documentation/config/core.txt | 35 +++++++++--- Makefile | 6 ++ builtin/fast-import.c | 2 +- builtin/index-pack.c | 4 +- builtin/pack-objects.c | 8 ++- bulk-checkin.c | 5 +- cache.h | 48 +++++++++++++++- commit-graph.c | 3 +- compat/mingw.h | 3 + compat/win32/flush.c | 28 +++++++++ config.c | 89 ++++++++++++++++++++++++++++- config.mak.uname | 3 + configure.ac | 8 +++ contrib/buildsystems/CMakeLists.txt | 3 +- csum-file.c | 5 +- csum-file.h | 3 +- environment.c | 3 +- git-compat-util.h | 24 ++++++++ midx.c | 3 +- object-file.c | 3 +- pack-bitmap-write.c | 3 +- pack-write.c | 13 +++-- read-cache.c | 19 ++++-- wrapper.c | 56 ++++++++++++++++++ write-or-die.c | 10 ++-- 25 files changed, 344 insertions(+), 43 deletions(-) create mode 100644 compat/win32/flush.c base-commit: abe6bb3905392d5eb6b01fa6e54d7e784e0522aa Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1093%2Fneerajsi-msft%2Fns%2Fcore-fsync-v2 Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1093/neerajsi-msft/ns/core-fsync-v2 Pull-Request: https://github.com/gitgitgadget/git/pull/1093 Range-diff vs v1: 1: 527380ddc3f ! 1: e79522cbdd4 fsync: add writeout-only mode for fsyncing repo data @@ Metadata Author: Neeraj Singh <neerajsi@microsoft.com> ## Commit message ## - fsync: add writeout-only mode for fsyncing repo data + core.fsyncmethod: add writeout-only mode + + This commit introduces the `core.fsyncmethod` configuration + knob, which can currently be set to `fsync` or `writeout-only`. The new writeout-only mode attempts to tell the operating system to flush its in-memory page cache to the storage hardware without issuing a @@ Documentation/config/core.txt: core.whitespace:: + using fsync and related primitives. ++ +* `fsync` uses the fsync() system call or platform equivalents. -+* `writeout-only` issues requests to send the writes to the storage -+ hardware, but does not send any FLUSH CACHE request. If the operating system -+ does not support the required interfaces, this falls back to fsync(). ++* `writeout-only` issues pagecache writeback requests, but depending on the ++ filesystem and storage hardware, data added to the repository may not be ++ durable in the event of a system crash. This is the default mode on macOS. + core.fsyncObjectFiles:: This boolean will enable 'fsync()' when writing object files. + + ## Makefile ## +@@ Makefile: all:: + # + # Define HAVE_CLOCK_MONOTONIC if your platform has CLOCK_MONOTONIC. + # ++# Define HAVE_SYNC_FILE_RANGE if your platform has sync_file_range. ++# + # Define NEEDS_LIBRT if your platform requires linking with librt (glibc version + # before 2.17) for clock_gettime and CLOCK_MONOTONIC. + # +@@ Makefile: ifdef HAVE_CLOCK_MONOTONIC + BASIC_CFLAGS += -DHAVE_CLOCK_MONOTONIC + endif + ++ifdef HAVE_SYNC_FILE_RANGE ++ BASIC_CFLAGS += -DHAVE_SYNC_FILE_RANGE ++endif ++ + ifdef NEEDS_LIBRT + EXTLIBS += -lrt + endif + ## cache.h ## @@ cache.h: extern int read_replace_refs; extern char *git_replace_ref_base; 2: 23311a10142 ! 2: ff80a94bf9a core.fsync: introduce granular fsync control @@ Commit message knob which can be used to control how components of the repository are made durable on disk. - This setting allows future extensibility of components - that could be synced in two ways: + This setting allows future extensibility of the list of + syncable components: * We issue a warning rather than an error for unrecognized components, so new configs can be used with old Git versions. * We support negation, so users can choose one of the default aggregate options and then remove components that they don't - want. + want. The user would then harden any new components added in + a Git version update. - This also support the common request of doing absolutely no + This also supports the common request of doing absolutely no fysncing with the `core.fsync=none` value, which is expected to make the test suite faster. - This commit introduces the new ability for the user to harden - the index, which is a requirement for being able to actually - find a file that has been added to the repo and then deleted - from the working tree. - Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> ## Documentation/config/core.txt ## @@ Documentation/config/core.txt: core.whitespace:: + hardened via the core.fsyncMethod when created or modified. You can + disable hardening of any component by prefixing it with a '-'. Later + items take precedence over earlier ones in the list. For example, -+ `core.fsync=all,-index` means "harden everything except the index". -+ Items that are not hardened may be lost in the event of an unclean -+ system shutdown. ++ `core.fsync=all,-pack-metadata` means "harden everything except pack ++ metadata." Items that are not hardened may be lost in the event of an ++ unclean system shutdown. ++ +* `none` disables fsync completely. This must be specified alone. +* `loose-object` hardens objects added to the repo in loose-object form. +* `pack` hardens objects added to the repo in packfile form. +* `pack-metadata` hardens packfile bitmaps and indexes. +* `commit-graph` hardens the commit graph file. -+* `index` hardens the index when it is modified. +* `objects` is an aggregate option that includes `loose-objects`, `pack`, + `pack-metadata`, and `commit-graph`. +* `default` is an aggregate option that is equivalent to `objects,-loose-object` @@ Documentation/config/core.txt: core.whitespace:: A value indicating the strategy Git will use to harden repository data using fsync and related primitives. @@ Documentation/config/core.txt: core.fsyncMethod:: - hardware, but does not send any FLUSH CACHE request. If the operating system - does not support the required interfaces, this falls back to fsync(). + filesystem and storage hardware, data added to the repository may not be + durable in the event of a system crash. This is the default mode on macOS. -core.fsyncObjectFiles:: - This boolean will enable 'fsync()' when writing object files. @@ builtin/fast-import.c: static void end_packfile(void) close_pack_windows(pack_data); - finalize_hashfile(pack_file, cur_pack_oid.hash, 0); -+ finalize_hashfile(pack_file, cur_pack_oid.hash, REPO_COMPONENT_PACK, 0); ++ finalize_hashfile(pack_file, cur_pack_oid.hash, FSYNC_COMPONENT_PACK, 0); fixup_pack_header_footer(pack_data->pack_fd, pack_data->hash, pack_data->pack_name, object_count, cur_pack_oid.hash, pack_size); @@ builtin/index-pack.c: static void conclude_pack(int fix_thin_pack, const char *c stop_progress_msg(&progress, msg.buf); strbuf_release(&msg); - finalize_hashfile(f, tail_hash, 0); -+ finalize_hashfile(f, tail_hash, REPO_COMPONENT_PACK, 0); ++ finalize_hashfile(f, tail_hash, FSYNC_COMPONENT_PACK, 0); hashcpy(read_hash, pack_hash); fixup_pack_header_footer(output_fd, pack_hash, curr_pack, nr_objects, @@ builtin/index-pack.c: static void final(const char *final_pack_name, const char close(input_fd); } else { - fsync_or_die(output_fd, curr_pack_name); -+ fsync_component_or_die(REPO_COMPONENT_PACK, output_fd, curr_pack_name); ++ fsync_component_or_die(FSYNC_COMPONENT_PACK, output_fd, curr_pack_name); err = close(output_fd); if (err) die_errno(_("error while closing pack file")); @@ builtin/pack-objects.c: static void write_pack_file(void) */ if (pack_to_stdout) { - finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_CLOSE); -+ finalize_hashfile(f, hash, REPO_COMPONENT_NONE, ++ finalize_hashfile(f, hash, FSYNC_COMPONENT_NONE, + CSUM_HASH_IN_STREAM | CSUM_CLOSE); } else if (nr_written == nr_remaining) { - finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); -+ finalize_hashfile(f, hash, REPO_COMPONENT_PACK, ++ finalize_hashfile(f, hash, FSYNC_COMPONENT_PACK, + CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); } else { - int fd = finalize_hashfile(f, hash, 0); -+ int fd = finalize_hashfile(f, hash, REPO_COMPONENT_PACK, 0); ++ int fd = finalize_hashfile(f, hash, FSYNC_COMPONENT_PACK, 0); fixup_pack_header_footer(fd, hash, pack_tmp_name, nr_written, hash, offset); close(fd); @@ bulk-checkin.c: static void finish_bulk_checkin(struct bulk_checkin_state *state goto clear_exit; } else if (state->nr_written == 1) { - finalize_hashfile(state->f, hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); -+ finalize_hashfile(state->f, hash, REPO_COMPONENT_PACK, ++ finalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK, + CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); } else { - int fd = finalize_hashfile(state->f, hash, 0); -+ int fd = finalize_hashfile(state->f, hash, REPO_COMPONENT_PACK, 0); ++ int fd = finalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK, 0); fixup_pack_header_footer(fd, hash, state->pack_tmp_name, state->nr_written, hash, state->offset); @@ cache.h: void reset_shared_repository(void); -extern int fsync_object_files; +/* + * These values are used to help identify parts of a repository to fsync. -+ * REPO_COMPONENT_NONE identifies data that will not be a persistent part of the ++ * FSYNC_COMPONENT_NONE identifies data that will not be a persistent part of the + * repository and so shouldn't be fsynced. + */ -+enum repo_component { -+ REPO_COMPONENT_NONE = 0, -+ REPO_COMPONENT_LOOSE_OBJECT = 1 << 0, -+ REPO_COMPONENT_PACK = 1 << 1, -+ REPO_COMPONENT_PACK_METADATA = 1 << 2, -+ REPO_COMPONENT_COMMIT_GRAPH = 1 << 3, -+ REPO_COMPONENT_INDEX = 1 << 4, ++enum fsync_component { ++ FSYNC_COMPONENT_NONE = 0, ++ FSYNC_COMPONENT_LOOSE_OBJECT = 1 << 0, ++ FSYNC_COMPONENT_PACK = 1 << 1, ++ FSYNC_COMPONENT_PACK_METADATA = 1 << 2, ++ FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3, +}; + -+#define FSYNC_COMPONENTS_DEFAULT (REPO_COMPONENT_PACK | \ -+ REPO_COMPONENT_PACK_METADATA | \ -+ REPO_COMPONENT_COMMIT_GRAPH) ++#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \ ++ FSYNC_COMPONENT_PACK_METADATA | \ ++ FSYNC_COMPONENT_COMMIT_GRAPH) + -+#define FSYNC_COMPONENTS_OBJECTS (REPO_COMPONENT_LOOSE_OBJECT | \ -+ REPO_COMPONENT_PACK | \ -+ REPO_COMPONENT_PACK_METADATA | \ -+ REPO_COMPONENT_COMMIT_GRAPH) ++#define FSYNC_COMPONENTS_OBJECTS (FSYNC_COMPONENT_LOOSE_OBJECT | \ ++ FSYNC_COMPONENT_PACK | \ ++ FSYNC_COMPONENT_PACK_METADATA | \ ++ FSYNC_COMPONENT_COMMIT_GRAPH) + -+#define FSYNC_COMPONENTS_ALL (REPO_COMPONENT_LOOSE_OBJECT | \ -+ REPO_COMPONENT_PACK | \ -+ REPO_COMPONENT_PACK_METADATA | \ -+ REPO_COMPONENT_COMMIT_GRAPH | \ -+ REPO_COMPONENT_INDEX) ++#define FSYNC_COMPONENTS_ALL (FSYNC_COMPONENT_LOOSE_OBJECT | \ ++ FSYNC_COMPONENT_PACK | \ ++ FSYNC_COMPONENT_PACK_METADATA | \ ++ FSYNC_COMPONENT_COMMIT_GRAPH) + + +/* + * A bitmask indicating which components of the repo should be fsynced. + */ -+extern enum repo_component fsync_components; ++extern enum fsync_component fsync_components; enum fsync_method { FSYNC_METHOD_FSYNC, @@ cache.h: int copy_file_with_time(const char *dst, const char *src, int mode); void write_or_die(int fd, const void *buf, size_t count); void fsync_or_die(int fd, const char *); -+inline void fsync_component_or_die(enum repo_component component, int fd, const char *msg) ++inline void fsync_component_or_die(enum fsync_component component, int fd, const char *msg) +{ + if (fsync_components & component) + fsync_or_die(fd, msg); @@ commit-graph.c: static int write_commit_graph_file(struct write_commit_graph_con close_commit_graph(ctx->r->objects); - finalize_hashfile(f, file_hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC); -+ finalize_hashfile(f, file_hash, REPO_COMPONENT_COMMIT_GRAPH, ++ finalize_hashfile(f, file_hash, FSYNC_COMPONENT_COMMIT_GRAPH, + CSUM_HASH_IN_STREAM | CSUM_FSYNC); free_chunkfile(cf); @@ config.c: static int git_parse_maybe_bool_text(const char *value) +static const struct fsync_component_entry { + const char *name; -+ enum repo_component component_bits; ++ enum fsync_component component_bits; +} fsync_component_table[] = { -+ { "loose-object", REPO_COMPONENT_LOOSE_OBJECT }, -+ { "pack", REPO_COMPONENT_PACK }, -+ { "pack-metadata", REPO_COMPONENT_PACK_METADATA }, -+ { "commit-graph", REPO_COMPONENT_COMMIT_GRAPH }, -+ { "index", REPO_COMPONENT_INDEX }, ++ { "loose-object", FSYNC_COMPONENT_LOOSE_OBJECT }, ++ { "pack", FSYNC_COMPONENT_PACK }, ++ { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA }, ++ { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, + { "objects", FSYNC_COMPONENTS_OBJECTS }, + { "default", FSYNC_COMPONENTS_DEFAULT }, + { "all", FSYNC_COMPONENTS_ALL }, +}; + -+static enum repo_component parse_fsync_components(const char *var, const char *string) ++static enum fsync_component parse_fsync_components(const char *var, const char *string) +{ -+ enum repo_component output = 0; ++ enum fsync_component output = 0; + + if (!strcmp(string, "none")) + return output; @@ csum-file.c: static void free_hashfile(struct hashfile *f) -int finalize_hashfile(struct hashfile *f, unsigned char *result, unsigned int flags) +int finalize_hashfile(struct hashfile *f, unsigned char *result, -+ enum repo_component component, unsigned int flags) ++ enum fsync_component component, unsigned int flags) { int fd; @@ csum-file.c: int finalize_hashfile(struct hashfile *f, unsigned char *result, un die_errno("%s: sha1 file error on close", f->name); ## csum-file.h ## +@@ + #ifndef CSUM_FILE_H + #define CSUM_FILE_H + ++#include "cache.h" + #include "hash.h" + + struct progress; @@ csum-file.h: int hashfile_truncate(struct hashfile *, struct hashfile_checkpoint *); struct hashfile *hashfd(int fd, const char *name); struct hashfile *hashfd_check(const char *name); struct hashfile *hashfd_throughput(int fd, const char *name, struct progress *tp); -int finalize_hashfile(struct hashfile *, unsigned char *, unsigned int); -+int finalize_hashfile(struct hashfile *, unsigned char *, enum repo_component, unsigned int); ++int finalize_hashfile(struct hashfile *, unsigned char *, enum fsync_component, unsigned int); void hashwrite(struct hashfile *, const void *, unsigned int); void hashflush(struct hashfile *f); void crc32_begin(struct hashfile *); @@ environment.c: const char *git_hooks_path; int zlib_compression_level = Z_BEST_SPEED; int pack_compression_level = Z_DEFAULT_COMPRESSION; enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT; -+enum repo_component fsync_components = FSYNC_COMPONENTS_DEFAULT; ++enum fsync_component fsync_components = FSYNC_COMPONENTS_DEFAULT; size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE; size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT; size_t delta_base_cache_limit = 96 * 1024 * 1024; @@ midx.c: static int write_midx_internal(const char *object_dir, write_chunkfile(cf, &ctx); - finalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM); -+ finalize_hashfile(f, midx_hash, REPO_COMPONENT_PACK_METADATA, ++ finalize_hashfile(f, midx_hash, FSYNC_COMPONENT_PACK_METADATA, + CSUM_FSYNC | CSUM_HASH_IN_STREAM); free_chunkfile(cf); @@ object-file.c: int hash_object_file(const struct git_hash_algo *algo, const void { - if (fsync_object_files) - fsync_or_die(fd, "loose object file"); -+ fsync_component_or_die(REPO_COMPONENT_LOOSE_OBJECT, fd, "loose object file"); ++ fsync_component_or_die(FSYNC_COMPONENT_LOOSE_OBJECT, fd, "loose object file"); if (close(fd) != 0) die_errno(_("error when closing loose object file")); } @@ pack-bitmap-write.c: void bitmap_writer_finish(struct pack_idx_entry **index, write_hash_cache(f, index, index_nr); - finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); -+ finalize_hashfile(f, NULL, REPO_COMPONENT_PACK_METADATA, ++ finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA, + CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); if (adjust_shared_perm(tmp_file.buf)) @@ pack-write.c: const char *write_idx_file(const char *index_name, struct pack_idx hashwrite(f, sha1, the_hash_algo->rawsz); - finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE | - ((opts->flags & WRITE_IDX_VERIFY) -+ finalize_hashfile(f, NULL, REPO_COMPONENT_PACK_METADATA, +- ? 0 : CSUM_FSYNC)); ++ finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA, + CSUM_HASH_IN_STREAM | CSUM_CLOSE | -+ ((opts->flags & WRITE_IDX_VERIFY) - ? 0 : CSUM_FSYNC)); ++ ((opts->flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC)); return index_name; } + @@ pack-write.c: const char *write_rev_file_order(const char *rev_name, if (rev_name && adjust_shared_perm(rev_name) < 0) die(_("failed to make %s readable"), rev_name); - finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE | - ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC)); -+ finalize_hashfile(f, NULL, REPO_COMPONENT_PACK_METADATA, ++ finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA, + CSUM_HASH_IN_STREAM | CSUM_CLOSE | + ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC)); @@ pack-write.c: void fixup_pack_header_footer(int pack_fd, the_hash_algo->final_fn(new_pack_hash, &new_hash_ctx); write_or_die(pack_fd, new_pack_hash, the_hash_algo->rawsz); - fsync_or_die(pack_fd, pack_name); -+ fsync_component_or_die(REPO_COMPONENT_PACK, pack_fd, pack_name); ++ fsync_component_or_die(FSYNC_COMPONENT_PACK, pack_fd, pack_name); } char *index_pack_lockfile(int ip_out, int *is_well_formed) ## read-cache.c ## -@@ read-cache.c: static int record_ieot(void) - * rely on it. - */ - static int do_write_index(struct index_state *istate, struct tempfile *tempfile, -- int strip_extensions) -+ int strip_extensions, unsigned flags) - { - uint64_t start = getnanotime(); - struct hashfile *f; -@@ read-cache.c: static int do_write_index(struct index_state *istate, struct tempfile *tempfile, - struct strbuf previous_name_buf = STRBUF_INIT, *previous_name; - int drop_cache_tree = istate->drop_cache_tree; - off_t offset; -+ int csum_fsync_flag; - int ieot_entries = 1; - struct index_entry_offset_table *ieot = NULL; - int nr, nr_threads; @@ read-cache.c: static int do_write_index(struct index_state *istate, struct tempfile *tempfile, return -1; } - finalize_hashfile(f, istate->oid.hash, CSUM_HASH_IN_STREAM); -+ csum_fsync_flag = 0; -+ if (!alternate_index_output && (flags & COMMIT_LOCK)) -+ csum_fsync_flag = CSUM_FSYNC; -+ -+ finalize_hashfile(f, istate->oid.hash, REPO_COMPONENT_INDEX, -+ CSUM_HASH_IN_STREAM | csum_fsync_flag); -+ ++ finalize_hashfile(f, istate->oid.hash, FSYNC_COMPONENT_NONE, CSUM_HASH_IN_STREAM); if (close_tempfile_gently(tempfile)) { error(_("could not close '%s'"), get_tempfile_path(tempfile)); return -1; -@@ read-cache.c: static int do_write_locked_index(struct index_state *istate, struct lock_file *l - */ - trace2_region_enter_printf("index", "do_write_index", the_repository, - "%s", get_lock_file_path(lock)); -- ret = do_write_index(istate, lock->tempfile, 0); -+ ret = do_write_index(istate, lock->tempfile, 0, flags); - trace2_region_leave_printf("index", "do_write_index", the_repository, - "%s", get_lock_file_path(lock)); - -@@ read-cache.c: static int clean_shared_index_files(const char *current_hex) - } - - static int write_shared_index(struct index_state *istate, -- struct tempfile **temp) -+ struct tempfile **temp, unsigned flags) - { - struct split_index *si = istate->split_index; - int ret, was_full = !istate->sparse_index; -@@ read-cache.c: static int write_shared_index(struct index_state *istate, - - trace2_region_enter_printf("index", "shared/do_write_index", - the_repository, "%s", get_tempfile_path(*temp)); -- ret = do_write_index(si->base, *temp, 1); -+ ret = do_write_index(si->base, *temp, 1, flags); - trace2_region_leave_printf("index", "shared/do_write_index", - the_repository, "%s", get_tempfile_path(*temp)); - -@@ read-cache.c: int write_locked_index(struct index_state *istate, struct lock_file *lock, - ret = do_write_locked_index(istate, lock, flags); - goto out; - } -- ret = write_shared_index(istate, &temp); -+ ret = write_shared_index(istate, &temp, flags); - - saved_errno = errno; - if (is_tempfile_active(temp)) -: ----------- > 3: 86e39b8f8d1 core.fsync: new option to harden the index -- gitgitgadget ^ permalink raw reply [flat|nested] 122+ messages in thread
* [PATCH v2 1/3] core.fsyncmethod: add writeout-only mode 2021-12-07 2:46 ` [PATCH v2 0/3] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget @ 2021-12-07 2:46 ` Neeraj Singh via GitGitGadget 2021-12-07 11:44 ` Patrick Steinhardt 2021-12-07 12:18 ` Ævar Arnfjörð Bjarmason 2021-12-07 2:46 ` [PATCH v2 2/3] core.fsync: introduce granular fsync control Neeraj Singh via GitGitGadget ` (3 subsequent siblings) 4 siblings, 2 replies; 122+ messages in thread From: Neeraj Singh via GitGitGadget @ 2021-12-07 2:46 UTC (permalink / raw) To: git Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps, Neeraj K. Singh, Neeraj Singh From: Neeraj Singh <neerajsi@microsoft.com> This commit introduces the `core.fsyncmethod` configuration knob, which can currently be set to `fsync` or `writeout-only`. The new writeout-only mode attempts to tell the operating system to flush its in-memory page cache to the storage hardware without issuing a CACHE_FLUSH command to the storage controller. Writeout-only fsync is significantly faster than a vanilla fsync on common hardware, since data is written to a disk-side cache rather than all the way to a durable medium. Later changes in this patch series will take advantage of this primitive to implement batching of hardware flushes. When git_fsync is called with FSYNC_WRITEOUT_ONLY, it may fail and the caller is expected to do an ordinary fsync as needed. On Apple platforms, the fsync system call does not issue a CACHE_FLUSH directive to the storage controller. This change updates fsync to do fcntl(F_FULLFSYNC) to make fsync actually durable. We maintain parity with existing behavior on Apple platforms by setting the default value of the new core.fsyncmethod option. Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> --- Documentation/config/core.txt | 9 +++++ Makefile | 6 ++++ cache.h | 7 ++++ compat/mingw.h | 3 ++ compat/win32/flush.c | 28 +++++++++++++++ config.c | 12 +++++++ config.mak.uname | 3 ++ configure.ac | 8 +++++ contrib/buildsystems/CMakeLists.txt | 3 +- environment.c | 2 +- git-compat-util.h | 24 +++++++++++++ wrapper.c | 56 +++++++++++++++++++++++++++++ write-or-die.c | 10 +++--- 13 files changed, 165 insertions(+), 6 deletions(-) create mode 100644 compat/win32/flush.c diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt index c04f62a54a1..dbb134f7136 100644 --- a/Documentation/config/core.txt +++ b/Documentation/config/core.txt @@ -547,6 +547,15 @@ core.whitespace:: is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent` errors. The default tab width is 8. Allowed values are 1 to 63. +core.fsyncMethod:: + A value indicating the strategy Git will use to harden repository data + using fsync and related primitives. ++ +* `fsync` uses the fsync() system call or platform equivalents. +* `writeout-only` issues pagecache writeback requests, but depending on the + filesystem and storage hardware, data added to the repository may not be + durable in the event of a system crash. This is the default mode on macOS. + core.fsyncObjectFiles:: This boolean will enable 'fsync()' when writing object files. + diff --git a/Makefile b/Makefile index d56c0e4aadc..cba024615c9 100644 --- a/Makefile +++ b/Makefile @@ -403,6 +403,8 @@ all:: # # Define HAVE_CLOCK_MONOTONIC if your platform has CLOCK_MONOTONIC. # +# Define HAVE_SYNC_FILE_RANGE if your platform has sync_file_range. +# # Define NEEDS_LIBRT if your platform requires linking with librt (glibc version # before 2.17) for clock_gettime and CLOCK_MONOTONIC. # @@ -1881,6 +1883,10 @@ ifdef HAVE_CLOCK_MONOTONIC BASIC_CFLAGS += -DHAVE_CLOCK_MONOTONIC endif +ifdef HAVE_SYNC_FILE_RANGE + BASIC_CFLAGS += -DHAVE_SYNC_FILE_RANGE +endif + ifdef NEEDS_LIBRT EXTLIBS += -lrt endif diff --git a/cache.h b/cache.h index eba12487b99..9cd60d94952 100644 --- a/cache.h +++ b/cache.h @@ -986,6 +986,13 @@ extern int read_replace_refs; extern char *git_replace_ref_base; extern int fsync_object_files; + +enum fsync_method { + FSYNC_METHOD_FSYNC, + FSYNC_METHOD_WRITEOUT_ONLY +}; + +extern enum fsync_method fsync_method; extern int core_preload_index; extern int precomposed_unicode; extern int protect_hfs; diff --git a/compat/mingw.h b/compat/mingw.h index c9a52ad64a6..6074a3d3ced 100644 --- a/compat/mingw.h +++ b/compat/mingw.h @@ -329,6 +329,9 @@ int mingw_getpagesize(void); #define getpagesize mingw_getpagesize #endif +int win32_fsync_no_flush(int fd); +#define fsync_no_flush win32_fsync_no_flush + struct rlimit { unsigned int rlim_cur; }; diff --git a/compat/win32/flush.c b/compat/win32/flush.c new file mode 100644 index 00000000000..75324c24ee7 --- /dev/null +++ b/compat/win32/flush.c @@ -0,0 +1,28 @@ +#include "../../git-compat-util.h" +#include <winternl.h> +#include "lazyload.h" + +int win32_fsync_no_flush(int fd) +{ + IO_STATUS_BLOCK io_status; + +#define FLUSH_FLAGS_FILE_DATA_ONLY 1 + + DECLARE_PROC_ADDR(ntdll.dll, NTSTATUS, NtFlushBuffersFileEx, + HANDLE FileHandle, ULONG Flags, PVOID Parameters, ULONG ParameterSize, + PIO_STATUS_BLOCK IoStatusBlock); + + if (!INIT_PROC_ADDR(NtFlushBuffersFileEx)) { + errno = ENOSYS; + return -1; + } + + memset(&io_status, 0, sizeof(io_status)); + if (NtFlushBuffersFileEx((HANDLE)_get_osfhandle(fd), FLUSH_FLAGS_FILE_DATA_ONLY, + NULL, 0, &io_status)) { + errno = EINVAL; + return -1; + } + + return 0; +} diff --git a/config.c b/config.c index c5873f3a706..c3410b8a868 100644 --- a/config.c +++ b/config.c @@ -1490,6 +1490,18 @@ static int git_default_core_config(const char *var, const char *value, void *cb) return 0; } + if (!strcmp(var, "core.fsyncmethod")) { + if (!value) + return config_error_nonbool(var); + if (!strcmp(value, "fsync")) + fsync_method = FSYNC_METHOD_FSYNC; + else if (!strcmp(value, "writeout-only")) + fsync_method = FSYNC_METHOD_WRITEOUT_ONLY; + else + warning(_("unknown %s value '%s'"), var, value); + + } + if (!strcmp(var, "core.fsyncobjectfiles")) { fsync_object_files = git_config_bool(var, value); return 0; diff --git a/config.mak.uname b/config.mak.uname index d0701f9beb0..774a09622d2 100644 --- a/config.mak.uname +++ b/config.mak.uname @@ -57,6 +57,7 @@ ifeq ($(uname_S),Linux) HAVE_CLOCK_MONOTONIC = YesPlease # -lrt is needed for clock_gettime on glibc <= 2.16 NEEDS_LIBRT = YesPlease + HAVE_SYNC_FILE_RANGE = YesPlease HAVE_GETDELIM = YesPlease FREAD_READS_DIRECTORIES = UnfortunatelyYes BASIC_CFLAGS += -DHAVE_SYSINFO @@ -453,6 +454,7 @@ endif CFLAGS = BASIC_CFLAGS = -nologo -I. -Icompat/vcbuild/include -DWIN32 -D_CONSOLE -DHAVE_STRING_H -D_CRT_SECURE_NO_WARNINGS -D_CRT_NONSTDC_NO_DEPRECATE COMPAT_OBJS = compat/msvc.o compat/winansi.o \ + compat/win32/flush.o \ compat/win32/path-utils.o \ compat/win32/pthread.o compat/win32/syslog.o \ compat/win32/trace2_win32_process_info.o \ @@ -628,6 +630,7 @@ ifeq ($(uname_S),MINGW) COMPAT_CFLAGS += -DSTRIP_EXTENSION=\".exe\" COMPAT_OBJS += compat/mingw.o compat/winansi.o \ compat/win32/trace2_win32_process_info.o \ + compat/win32/flush.o \ compat/win32/path-utils.o \ compat/win32/pthread.o compat/win32/syslog.o \ compat/win32/dirent.o diff --git a/configure.ac b/configure.ac index 5ee25ec95c8..6bd6bef1c44 100644 --- a/configure.ac +++ b/configure.ac @@ -1082,6 +1082,14 @@ AC_COMPILE_IFELSE([CLOCK_MONOTONIC_SRC], [AC_MSG_RESULT([no]) HAVE_CLOCK_MONOTONIC=]) GIT_CONF_SUBST([HAVE_CLOCK_MONOTONIC]) + +# +# Define HAVE_SYNC_FILE_RANGE=YesPlease if sync_file_range is available. +GIT_CHECK_FUNC(sync_file_range, + [HAVE_SYNC_FILE_RANGE=YesPlease], + [HAVE_SYNC_FILE_RANGE]) +GIT_CONF_SUBST([HAVE_SYNC_FILE_RANGE]) + # # Define NO_SETITIMER if you don't have setitimer. GIT_CHECK_FUNC(setitimer, diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt index 86b46114464..6d7bc16d054 100644 --- a/contrib/buildsystems/CMakeLists.txt +++ b/contrib/buildsystems/CMakeLists.txt @@ -261,7 +261,8 @@ if(CMAKE_SYSTEM_NAME STREQUAL "Windows") NOGDI OBJECT_CREATION_MODE=1 __USE_MINGW_ANSI_STDIO=0 USE_NED_ALLOCATOR OVERRIDE_STRDUP MMAP_PREVENTS_DELETE USE_WIN32_MMAP UNICODE _UNICODE HAVE_WPGMPTR ENSURE_MSYSTEM_IS_SET) - list(APPEND compat_SOURCES compat/mingw.c compat/winansi.c compat/win32/path-utils.c + list(APPEND compat_SOURCES compat/mingw.c compat/winansi.c + compat/win32/flush.c compat/win32/path-utils.c compat/win32/pthread.c compat/win32mmap.c compat/win32/syslog.c compat/win32/trace2_win32_process_info.c compat/win32/dirent.c compat/nedmalloc/nedmalloc.c compat/strdup.c) diff --git a/environment.c b/environment.c index 9da7f3c1a19..f9140e842cf 100644 --- a/environment.c +++ b/environment.c @@ -41,7 +41,7 @@ const char *git_attributes_file; const char *git_hooks_path; int zlib_compression_level = Z_BEST_SPEED; int pack_compression_level = Z_DEFAULT_COMPRESSION; -int fsync_object_files; +enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT; size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE; size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT; size_t delta_base_cache_limit = 96 * 1024 * 1024; diff --git a/git-compat-util.h b/git-compat-util.h index c6bd2a84e55..cb9abd7a08c 100644 --- a/git-compat-util.h +++ b/git-compat-util.h @@ -1239,6 +1239,30 @@ __attribute__((format (printf, 1, 2))) NORETURN void BUG(const char *fmt, ...); #endif +#ifdef __APPLE__ +#define FSYNC_METHOD_DEFAULT FSYNC_METHOD_WRITEOUT_ONLY +#else +#define FSYNC_METHOD_DEFAULT FSYNC_METHOD_FSYNC +#endif + +enum fsync_action { + FSYNC_WRITEOUT_ONLY, + FSYNC_HARDWARE_FLUSH +}; + +/* + * Issues an fsync against the specified file according to the specified mode. + * + * FSYNC_WRITEOUT_ONLY attempts to use interfaces available on some operating + * systems to flush the OS cache without issuing a flush command to the storage + * controller. If those interfaces are unavailable, the function fails with + * ENOSYS. + * + * FSYNC_HARDWARE_FLUSH does an OS writeout and hardware flush to ensure that + * changes are durable. It is not expected to fail. + */ +int git_fsync(int fd, enum fsync_action action); + /* * Preserves errno, prints a message, but gives no warning for ENOENT. * Returns 0 on success, which includes trying to unlink an object that does diff --git a/wrapper.c b/wrapper.c index 36e12119d76..1c5f2c87791 100644 --- a/wrapper.c +++ b/wrapper.c @@ -546,6 +546,62 @@ int xmkstemp_mode(char *filename_template, int mode) return fd; } +int git_fsync(int fd, enum fsync_action action) +{ + switch (action) { + case FSYNC_WRITEOUT_ONLY: + +#ifdef __APPLE__ + /* + * on macOS, fsync just causes filesystem cache writeback but does not + * flush hardware caches. + */ + return fsync(fd); +#endif + +#ifdef HAVE_SYNC_FILE_RANGE + /* + * On linux 2.6.17 and above, sync_file_range is the way to issue + * a writeback without a hardware flush. An offset of 0 and size of 0 + * indicates writeout of the entire file and the wait flags ensure that all + * dirty data is written to the disk (potentially in a disk-side cache) + * before we continue. + */ + + return sync_file_range(fd, 0, 0, SYNC_FILE_RANGE_WAIT_BEFORE | + SYNC_FILE_RANGE_WRITE | + SYNC_FILE_RANGE_WAIT_AFTER); +#endif + +#ifdef fsync_no_flush + return fsync_no_flush(fd); +#endif + + errno = ENOSYS; + return -1; + + case FSYNC_HARDWARE_FLUSH: + /* + * On some platforms fsync may return EINTR. Try again in this + * case, since callers asking for a hardware flush may die if + * this function returns an error. + */ + for (;;) { + int err; +#ifdef __APPLE__ + err = fcntl(fd, F_FULLFSYNC); +#else + err = fsync(fd); +#endif + if (err >= 0 || errno != EINTR) + return err; + } + + default: + BUG("unexpected git_fsync(%d) call", action); + } +} + static int warn_if_unremovable(const char *op, const char *file, int rc) { int err; diff --git a/write-or-die.c b/write-or-die.c index 0b1ec8190b6..0702acdd5e8 100644 --- a/write-or-die.c +++ b/write-or-die.c @@ -57,10 +57,12 @@ void fprintf_or_die(FILE *f, const char *fmt, ...) void fsync_or_die(int fd, const char *msg) { - while (fsync(fd) < 0) { - if (errno != EINTR) - die_errno("fsync error on '%s'", msg); - } + if (fsync_method == FSYNC_METHOD_WRITEOUT_ONLY && + git_fsync(fd, FSYNC_WRITEOUT_ONLY) >= 0) + return; + + if (git_fsync(fd, FSYNC_HARDWARE_FLUSH) < 0) + die_errno("fsync error on '%s'", msg); } void write_or_die(int fd, const void *buf, size_t count) -- gitgitgadget ^ permalink raw reply related [flat|nested] 122+ messages in thread
* Re: [PATCH v2 1/3] core.fsyncmethod: add writeout-only mode 2021-12-07 2:46 ` [PATCH v2 1/3] core.fsyncmethod: add writeout-only mode Neeraj Singh via GitGitGadget @ 2021-12-07 11:44 ` Patrick Steinhardt 2021-12-07 12:14 ` Ævar Arnfjörð Bjarmason 2021-12-07 23:29 ` Neeraj Singh 2021-12-07 12:18 ` Ævar Arnfjörð Bjarmason 1 sibling, 2 replies; 122+ messages in thread From: Patrick Steinhardt @ 2021-12-07 11:44 UTC (permalink / raw) To: Neeraj Singh via GitGitGadget Cc: git, rsbecker, bagasdotme, newren, avarab, nksingh85, Neeraj K. Singh [-- Attachment #1: Type: text/plain, Size: 4326 bytes --] On Tue, Dec 07, 2021 at 02:46:49AM +0000, Neeraj Singh via GitGitGadget wrote: > From: Neeraj Singh <neerajsi@microsoft.com> [snip] > --- a/compat/mingw.h > +++ b/compat/mingw.h > @@ -329,6 +329,9 @@ int mingw_getpagesize(void); > #define getpagesize mingw_getpagesize > #endif > > +int win32_fsync_no_flush(int fd); > +#define fsync_no_flush win32_fsync_no_flush > + > struct rlimit { > unsigned int rlim_cur; > }; > diff --git a/compat/win32/flush.c b/compat/win32/flush.c > new file mode 100644 > index 00000000000..75324c24ee7 > --- /dev/null > +++ b/compat/win32/flush.c > @@ -0,0 +1,28 @@ > +#include "../../git-compat-util.h" > +#include <winternl.h> > +#include "lazyload.h" > + > +int win32_fsync_no_flush(int fd) > +{ > + IO_STATUS_BLOCK io_status; > + > +#define FLUSH_FLAGS_FILE_DATA_ONLY 1 > + > + DECLARE_PROC_ADDR(ntdll.dll, NTSTATUS, NtFlushBuffersFileEx, > + HANDLE FileHandle, ULONG Flags, PVOID Parameters, ULONG ParameterSize, > + PIO_STATUS_BLOCK IoStatusBlock); > + > + if (!INIT_PROC_ADDR(NtFlushBuffersFileEx)) { > + errno = ENOSYS; > + return -1; > + } I'm wondering whether it would make sense to fall back to fsync(3P) in case we cannot use writeout-only, but I see that were doing essentially that in `fsync_or_die()`. There is no indicator to the user though that writeout-only doesn't work -- do we want to print a one-time warning? > + memset(&io_status, 0, sizeof(io_status)); > + if (NtFlushBuffersFileEx((HANDLE)_get_osfhandle(fd), FLUSH_FLAGS_FILE_DATA_ONLY, > + NULL, 0, &io_status)) { > + errno = EINVAL; > + return -1; > + } > + > + return 0; > +} [snip] > diff --git a/wrapper.c b/wrapper.c > index 36e12119d76..1c5f2c87791 100644 > --- a/wrapper.c > +++ b/wrapper.c > @@ -546,6 +546,62 @@ int xmkstemp_mode(char *filename_template, int mode) > return fd; > } > > +int git_fsync(int fd, enum fsync_action action) > +{ > + switch (action) { > + case FSYNC_WRITEOUT_ONLY: > + > +#ifdef __APPLE__ > + /* > + * on macOS, fsync just causes filesystem cache writeback but does not > + * flush hardware caches. > + */ > + return fsync(fd); Below we're looping around `EINTR` -- are Apple systems never returning it? Patrick > +#endif > + > +#ifdef HAVE_SYNC_FILE_RANGE > + /* > + * On linux 2.6.17 and above, sync_file_range is the way to issue > + * a writeback without a hardware flush. An offset of 0 and size of 0 > + * indicates writeout of the entire file and the wait flags ensure that all > + * dirty data is written to the disk (potentially in a disk-side cache) > + * before we continue. > + */ > + > + return sync_file_range(fd, 0, 0, SYNC_FILE_RANGE_WAIT_BEFORE | > + SYNC_FILE_RANGE_WRITE | > + SYNC_FILE_RANGE_WAIT_AFTER); > +#endif > + > +#ifdef fsync_no_flush > + return fsync_no_flush(fd); > +#endif > + > + errno = ENOSYS; > + return -1; > + > + case FSYNC_HARDWARE_FLUSH: > + /* > + * On some platforms fsync may return EINTR. Try again in this > + * case, since callers asking for a hardware flush may die if > + * this function returns an error. > + */ > + for (;;) { > + int err; > +#ifdef __APPLE__ > + err = fcntl(fd, F_FULLFSYNC); > +#else > + err = fsync(fd); > +#endif > + if (err >= 0 || errno != EINTR) > + return err; > + } > + > + default: > + BUG("unexpected git_fsync(%d) call", action); > + } > +} > + > static int warn_if_unremovable(const char *op, const char *file, int rc) > { > int err; > diff --git a/write-or-die.c b/write-or-die.c > index 0b1ec8190b6..0702acdd5e8 100644 > --- a/write-or-die.c > +++ b/write-or-die.c > @@ -57,10 +57,12 @@ void fprintf_or_die(FILE *f, const char *fmt, ...) > > void fsync_or_die(int fd, const char *msg) > { > - while (fsync(fd) < 0) { > - if (errno != EINTR) > - die_errno("fsync error on '%s'", msg); > - } > + if (fsync_method == FSYNC_METHOD_WRITEOUT_ONLY && > + git_fsync(fd, FSYNC_WRITEOUT_ONLY) >= 0) > + return; > + > + if (git_fsync(fd, FSYNC_HARDWARE_FLUSH) < 0) > + die_errno("fsync error on '%s'", msg); > } > > void write_or_die(int fd, const void *buf, size_t count) > -- > gitgitgadget > [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v2 1/3] core.fsyncmethod: add writeout-only mode 2021-12-07 11:44 ` Patrick Steinhardt @ 2021-12-07 12:14 ` Ævar Arnfjörð Bjarmason 2021-12-07 23:29 ` Neeraj Singh 1 sibling, 0 replies; 122+ messages in thread From: Ævar Arnfjörð Bjarmason @ 2021-12-07 12:14 UTC (permalink / raw) To: Patrick Steinhardt Cc: Neeraj Singh via GitGitGadget, git, rsbecker, bagasdotme, newren, nksingh85, Neeraj K. Singh On Tue, Dec 07 2021, Patrick Steinhardt wrote: > [[PGP Signed Part:Undecided]] > On Tue, Dec 07, 2021 at 02:46:49AM +0000, Neeraj Singh via GitGitGadget wrote: >> From: Neeraj Singh <neerajsi@microsoft.com> > [...] > [snip] >> diff --git a/wrapper.c b/wrapper.c >> index 36e12119d76..1c5f2c87791 100644 >> --- a/wrapper.c >> +++ b/wrapper.c >> @@ -546,6 +546,62 @@ int xmkstemp_mode(char *filename_template, int mode) >> return fd; >> } >> >> +int git_fsync(int fd, enum fsync_action action) >> +{ >> + switch (action) { >> + case FSYNC_WRITEOUT_ONLY: >> + >> +#ifdef __APPLE__ >> + /* >> + * on macOS, fsync just causes filesystem cache writeback but does not >> + * flush hardware caches. >> + */ >> + return fsync(fd); > > Below we're looping around `EINTR` -- are Apple systems never returning > it? I think so per cccdfd22436 (fsync(): be prepared to see EINTR, 2021-06-04), but I'm not sure, but in any case it would make sense for this to just call the same loop we've been calling elsewhere. It doesn't seem to hurt, so we can just do that part in the portable portion of the code. ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v2 1/3] core.fsyncmethod: add writeout-only mode 2021-12-07 11:44 ` Patrick Steinhardt 2021-12-07 12:14 ` Ævar Arnfjörð Bjarmason @ 2021-12-07 23:29 ` Neeraj Singh 1 sibling, 0 replies; 122+ messages in thread From: Neeraj Singh @ 2021-12-07 23:29 UTC (permalink / raw) To: Patrick Steinhardt Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Ævar Arnfjörð Bjarmason, Neeraj K. Singh On Tue, Dec 7, 2021 at 3:45 AM Patrick Steinhardt <ps@pks.im> wrote: > > On Tue, Dec 07, 2021 at 02:46:49AM +0000, Neeraj Singh via GitGitGadget wrote: > > From: Neeraj Singh <neerajsi@microsoft.com> > [snip] > > --- a/compat/mingw.h > > +++ b/compat/mingw.h > > @@ -329,6 +329,9 @@ int mingw_getpagesize(void); > > #define getpagesize mingw_getpagesize > > #endif > > > > +int win32_fsync_no_flush(int fd); > > +#define fsync_no_flush win32_fsync_no_flush > > + > > struct rlimit { > > unsigned int rlim_cur; > > }; > > diff --git a/compat/win32/flush.c b/compat/win32/flush.c > > new file mode 100644 > > index 00000000000..75324c24ee7 > > --- /dev/null > > +++ b/compat/win32/flush.c > > @@ -0,0 +1,28 @@ > > +#include "../../git-compat-util.h" > > +#include <winternl.h> > > +#include "lazyload.h" > > + > > +int win32_fsync_no_flush(int fd) > > +{ > > + IO_STATUS_BLOCK io_status; > > + > > +#define FLUSH_FLAGS_FILE_DATA_ONLY 1 > > + > > + DECLARE_PROC_ADDR(ntdll.dll, NTSTATUS, NtFlushBuffersFileEx, > > + HANDLE FileHandle, ULONG Flags, PVOID Parameters, ULONG ParameterSize, > > + PIO_STATUS_BLOCK IoStatusBlock); > > + > > + if (!INIT_PROC_ADDR(NtFlushBuffersFileEx)) { > > + errno = ENOSYS; > > + return -1; > > + } > > I'm wondering whether it would make sense to fall back to fsync(3P) in > case we cannot use writeout-only, but I see that were doing essentially > that in `fsync_or_die()`. There is no indicator to the user though that > writeout-only doesn't work -- do we want to print a one-time warning? > I wanted to leave the fallback to the caller so that the algorithm can be adjusted in some way based on whether writeout-only succeeded. For batched fsync object files, we refrain from doing the last fsync if we were doing real fsyncs all along. I didn't want to issue a warning, since this writeout-only codepath might be invoked from multiple subprocesses, which would each potentially issue their one warning. The consequence of failing writeout only is worse performance, but should not be compromised safety, so I'm not sure the user gets enough from the warning to justify something that's potentially spammy. In practice, when batch mode is adopted on Windows (by default), some older pre-Win8 systems will experience fsyncs and equivalent performance to what they're seeing today. I don't want these users to have a warning too. > > + memset(&io_status, 0, sizeof(io_status)); > > + if (NtFlushBuffersFileEx((HANDLE)_get_osfhandle(fd), FLUSH_FLAGS_FILE_DATA_ONLY, > > + NULL, 0, &io_status)) { > > + errno = EINVAL; > > + return -1; > > + } > > + > > + return 0; > > +} > > [snip] > > diff --git a/wrapper.c b/wrapper.c > > index 36e12119d76..1c5f2c87791 100644 > > --- a/wrapper.c > > +++ b/wrapper.c > > @@ -546,6 +546,62 @@ int xmkstemp_mode(char *filename_template, int mode) > > return fd; > > } > > > > +int git_fsync(int fd, enum fsync_action action) > > +{ > > + switch (action) { > > + case FSYNC_WRITEOUT_ONLY: > > + > > +#ifdef __APPLE__ > > + /* > > + * on macOS, fsync just causes filesystem cache writeback but does not > > + * flush hardware caches. > > + */ > > + return fsync(fd); > > Below we're looping around `EINTR` -- are Apple systems never returning > it? > The EINTR check was added due to a test failure on HP NonStop. I don't believe any other platform has actually complained about that. Thanks again for the code review! -Neeraj ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v2 1/3] core.fsyncmethod: add writeout-only mode 2021-12-07 2:46 ` [PATCH v2 1/3] core.fsyncmethod: add writeout-only mode Neeraj Singh via GitGitGadget 2021-12-07 11:44 ` Patrick Steinhardt @ 2021-12-07 12:18 ` Ævar Arnfjörð Bjarmason 2021-12-07 23:58 ` Neeraj Singh 1 sibling, 1 reply; 122+ messages in thread From: Ævar Arnfjörð Bjarmason @ 2021-12-07 12:18 UTC (permalink / raw) To: Neeraj Singh via GitGitGadget Cc: git, rsbecker, bagasdotme, newren, nksingh85, ps, Neeraj Singh On Tue, Dec 07 2021, Neeraj Singh via GitGitGadget wrote: > From: Neeraj Singh <neerajsi@microsoft.com> > > This commit introduces the `core.fsyncmethod` configuration Just a commit msg nit: core.fsyncMethod (I see the docs etc. are using it camelCased, good.. > diff --git a/compat/win32/flush.c b/compat/win32/flush.c > new file mode 100644 > index 00000000000..75324c24ee7 > --- /dev/null > +++ b/compat/win32/flush.c > @@ -0,0 +1,28 @@ > +#include "../../git-compat-util.h" nit: Just FWIW I think the better thing is '#include "git-compat-util.h"', i.e. we're compiling at the top-level and have added it to -I. (I know a lot of compat/ and contrib/ and even main-tree stuff does that, but just FWIW it's not needed). > + if (!strcmp(var, "core.fsyncmethod")) { > + if (!value) > + return config_error_nonbool(var); > + if (!strcmp(value, "fsync")) > + fsync_method = FSYNC_METHOD_FSYNC; > + else if (!strcmp(value, "writeout-only")) > + fsync_method = FSYNC_METHOD_WRITEOUT_ONLY; > + else As a non-nit comment I think this config schema looks great so far. > + warning(_("unknown %s value '%s'"), var, value); Just a suggestion maybe something slightly scarier like: "unknown core.fsyncMethod value '%s'; config from future git version? ignoring requested fsync strategy" Also using the nicer camelCased version instead of "var" (also helps translators with context...) > +int git_fsync(int fd, enum fsync_action action) > +{ > + switch (action) { > + case FSYNC_WRITEOUT_ONLY: > + > +#ifdef __APPLE__ > + /* > + * on macOS, fsync just causes filesystem cache writeback but does not > + * flush hardware caches. > + */ > + return fsync(fd); > +#endif > + > +#ifdef HAVE_SYNC_FILE_RANGE > + /* > + * On linux 2.6.17 and above, sync_file_range is the way to issue > + * a writeback without a hardware flush. An offset of 0 and size of 0 > + * indicates writeout of the entire file and the wait flags ensure that all > + * dirty data is written to the disk (potentially in a disk-side cache) > + * before we continue. > + */ > + > + return sync_file_range(fd, 0, 0, SYNC_FILE_RANGE_WAIT_BEFORE | > + SYNC_FILE_RANGE_WRITE | > + SYNC_FILE_RANGE_WAIT_AFTER); > +#endif > + > +#ifdef fsync_no_flush > + return fsync_no_flush(fd); > +#endif > + > + errno = ENOSYS; > + return -1; > + > + case FSYNC_HARDWARE_FLUSH: > + /* > + * On some platforms fsync may return EINTR. Try again in this > + * case, since callers asking for a hardware flush may die if > + * this function returns an error. > + */ > + for (;;) { > + int err; > +#ifdef __APPLE__ > + err = fcntl(fd, F_FULLFSYNC); > +#else > + err = fsync(fd); > +#endif > + if (err >= 0 || errno != EINTR) > + return err; > + } > + > + default: > + BUG("unexpected git_fsync(%d) call", action); Don't include such "default" cases, you have an exhaustive "enum", if you skip it the compiler will check this for you. > + } > +} > + > static int warn_if_unremovable(const char *op, const char *file, int rc) Just a code nit: I think it's very much preferred if possible to have as much of code like this compile on all platforms. See the series at 4002e87cb25 (grep: remove #ifdef NO_PTHREADS, 2018-11-03) is part of for a good example. Maybe not worth it in this case since they're not nested ifdef's. I'm basically thinking of something (also re Patrick's comment on the 2nd patch) where we have a platform_fsync() whose return value/arguments/whatever capture this "I want to return now" or "you should be looping" and takes the enum_fsync_action" strategy. Then the git_fsync() would be the platform-independent looping etc., and another funciton would do the "one fsync at a time, maybe call me again". Maybe it would suck more, just food for thought... :) ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v2 1/3] core.fsyncmethod: add writeout-only mode 2021-12-07 12:18 ` Ævar Arnfjörð Bjarmason @ 2021-12-07 23:58 ` Neeraj Singh 0 siblings, 0 replies; 122+ messages in thread From: Neeraj Singh @ 2021-12-07 23:58 UTC (permalink / raw) To: Ævar Arnfjörð Bjarmason Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Patrick Steinhardt, Neeraj Singh On Tue, Dec 7, 2021 at 4:27 AM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote: > > > On Tue, Dec 07 2021, Neeraj Singh via GitGitGadget wrote: > > > From: Neeraj Singh <neerajsi@microsoft.com> > > > > This commit introduces the `core.fsyncmethod` configuration > > Just a commit msg nit: core.fsyncMethod (I see the docs etc. are using > it camelCased, good.. Will fix. > > diff --git a/compat/win32/flush.c b/compat/win32/flush.c > > new file mode 100644 > > index 00000000000..75324c24ee7 > > --- /dev/null > > +++ b/compat/win32/flush.c > > @@ -0,0 +1,28 @@ > > +#include "../../git-compat-util.h" > > nit: Just FWIW I think the better thing is '#include > "git-compat-util.h"', i.e. we're compiling at the top-level and have > added it to -I. > > (I know a lot of compat/ and contrib/ and even main-tree stuff does > that, but just FWIW it's not needed). > Will fix. > > + if (!strcmp(var, "core.fsyncmethod")) { > > + if (!value) > > + return config_error_nonbool(var); > > + if (!strcmp(value, "fsync")) > > + fsync_method = FSYNC_METHOD_FSYNC; > > + else if (!strcmp(value, "writeout-only")) > > + fsync_method = FSYNC_METHOD_WRITEOUT_ONLY; > > + else > > As a non-nit comment I think this config schema looks great so far. > > > + warning(_("unknown %s value '%s'"), var, value); > > Just a suggestion maybe something slightly scarier like: > > "unknown core.fsyncMethod value '%s'; config from future git version? ignoring requested fsync strategy" > > Also using the nicer camelCased version instead of "var" (also helps > translators with context...) > Will fix. The motivation for this scheme was to 'factor' the messages so there would be less to translate. But I see now that the message doesn't have enough context to translate reasonably. > > +int git_fsync(int fd, enum fsync_action action) > > +{ > > + switch (action) { > > + case FSYNC_WRITEOUT_ONLY: > > + > > +#ifdef __APPLE__ > > + /* > > + * on macOS, fsync just causes filesystem cache writeback but does not > > + * flush hardware caches. > > + */ > > + return fsync(fd); > > +#endif > > + > > +#ifdef HAVE_SYNC_FILE_RANGE > > + /* > > + * On linux 2.6.17 and above, sync_file_range is the way to issue > > + * a writeback without a hardware flush. An offset of 0 and size of 0 > > + * indicates writeout of the entire file and the wait flags ensure that all > > + * dirty data is written to the disk (potentially in a disk-side cache) > > + * before we continue. > > + */ > > + > > + return sync_file_range(fd, 0, 0, SYNC_FILE_RANGE_WAIT_BEFORE | > > + SYNC_FILE_RANGE_WRITE | > > + SYNC_FILE_RANGE_WAIT_AFTER); > > +#endif > > + > > +#ifdef fsync_no_flush > > + return fsync_no_flush(fd); > > +#endif > > + > > + errno = ENOSYS; > > + return -1; > > + > > + case FSYNC_HARDWARE_FLUSH: > > + /* > > + * On some platforms fsync may return EINTR. Try again in this > > + * case, since callers asking for a hardware flush may die if > > + * this function returns an error. > > + */ > > + for (;;) { > > + int err; > > +#ifdef __APPLE__ > > + err = fcntl(fd, F_FULLFSYNC); > > +#else > > + err = fsync(fd); > > +#endif > > + if (err >= 0 || errno != EINTR) > > + return err; > > + } > > + > > + default: > > + BUG("unexpected git_fsync(%d) call", action); > > Don't include such "default" cases, you have an exhaustive "enum", if > you skip it the compiler will check this for you. > Junio gave the feedback to include this "default:" case in the switch [1]. Removing the default leads to the "error: control reaches end of non-void function" on gcc. Fixing that error and adding a trial option does give the exhaustiveness error that you're talking about. I'd rather just leave this as-is since the BUG() obviates the need for an in-practice-unreachable return statement. [1] https://lore.kernel.org/git/xmqqfsu70x58.fsf@gitster.g/ > > + } > > +} > > + > > static int warn_if_unremovable(const char *op, const char *file, int rc) > > Just a code nit: I think it's very much preferred if possible to have as > much of code like this compile on all platforms. See the series at > 4002e87cb25 (grep: remove #ifdef NO_PTHREADS, 2018-11-03) is part of for > a good example. > > Maybe not worth it in this case since they're not nested ifdef's. > > I'm basically thinking of something (also re Patrick's comment on the > 2nd patch) where we have a platform_fsync() whose return > value/arguments/whatever capture this "I want to return now" or "you > should be looping" and takes the enum_fsync_action" strategy. > > Then the git_fsync() would be the platform-independent looping etc., and > another funciton would do the "one fsync at a time, maybe call me > again". > > Maybe it would suck more, just food for thought... :) I'm going to introduce a new static function called fsync_loop which does the looping and I'll call it from git_fsync. That appears to be the cleanest to me to address your and Patrick's feedback. Thanks, Neeraj ^ permalink raw reply [flat|nested] 122+ messages in thread
* [PATCH v2 2/3] core.fsync: introduce granular fsync control 2021-12-07 2:46 ` [PATCH v2 0/3] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget 2021-12-07 2:46 ` [PATCH v2 1/3] core.fsyncmethod: add writeout-only mode Neeraj Singh via GitGitGadget @ 2021-12-07 2:46 ` Neeraj Singh via GitGitGadget 2021-12-07 11:53 ` Patrick Steinhardt 2021-12-07 12:29 ` Ævar Arnfjörð Bjarmason 2021-12-07 2:46 ` [PATCH v2 3/3] core.fsync: new option to harden the index Neeraj Singh via GitGitGadget ` (2 subsequent siblings) 4 siblings, 2 replies; 122+ messages in thread From: Neeraj Singh via GitGitGadget @ 2021-12-07 2:46 UTC (permalink / raw) To: git Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps, Neeraj K. Singh, Neeraj Singh From: Neeraj Singh <neerajsi@microsoft.com> This commit introduces the `core.fsync` configuration knob which can be used to control how components of the repository are made durable on disk. This setting allows future extensibility of the list of syncable components: * We issue a warning rather than an error for unrecognized components, so new configs can be used with old Git versions. * We support negation, so users can choose one of the default aggregate options and then remove components that they don't want. The user would then harden any new components added in a Git version update. This also supports the common request of doing absolutely no fysncing with the `core.fsync=none` value, which is expected to make the test suite faster. Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> --- Documentation/config/core.txt | 27 +++++++++---- builtin/fast-import.c | 2 +- builtin/index-pack.c | 4 +- builtin/pack-objects.c | 8 ++-- bulk-checkin.c | 5 ++- cache.h | 39 +++++++++++++++++- commit-graph.c | 3 +- config.c | 76 ++++++++++++++++++++++++++++++++++- csum-file.c | 5 ++- csum-file.h | 3 +- environment.c | 1 + midx.c | 3 +- object-file.c | 3 +- pack-bitmap-write.c | 3 +- pack-write.c | 13 +++--- read-cache.c | 2 +- 16 files changed, 164 insertions(+), 33 deletions(-) diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt index dbb134f7136..4f1747ec871 100644 --- a/Documentation/config/core.txt +++ b/Documentation/config/core.txt @@ -547,6 +547,25 @@ core.whitespace:: is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent` errors. The default tab width is 8. Allowed values are 1 to 63. +core.fsync:: + A comma-separated list of parts of the repository which should be + hardened via the core.fsyncMethod when created or modified. You can + disable hardening of any component by prefixing it with a '-'. Later + items take precedence over earlier ones in the list. For example, + `core.fsync=all,-pack-metadata` means "harden everything except pack + metadata." Items that are not hardened may be lost in the event of an + unclean system shutdown. ++ +* `none` disables fsync completely. This must be specified alone. +* `loose-object` hardens objects added to the repo in loose-object form. +* `pack` hardens objects added to the repo in packfile form. +* `pack-metadata` hardens packfile bitmaps and indexes. +* `commit-graph` hardens the commit graph file. +* `objects` is an aggregate option that includes `loose-objects`, `pack`, + `pack-metadata`, and `commit-graph`. +* `default` is an aggregate option that is equivalent to `objects,-loose-object` +* `all` is an aggregate option that syncs all individual components above. + core.fsyncMethod:: A value indicating the strategy Git will use to harden repository data using fsync and related primitives. @@ -556,14 +575,6 @@ core.fsyncMethod:: filesystem and storage hardware, data added to the repository may not be durable in the event of a system crash. This is the default mode on macOS. -core.fsyncObjectFiles:: - This boolean will enable 'fsync()' when writing object files. -+ -This is a total waste of time and effort on a filesystem that orders -data writes properly, but can be useful for filesystems that do not use -journalling (traditional UNIX filesystems) or that only journal metadata -and not file contents (OS X's HFS+, or Linux ext3 with "data=writeback"). - core.preloadIndex:: Enable parallel index preload for operations like 'git diff' + diff --git a/builtin/fast-import.c b/builtin/fast-import.c index 20406f67754..e27a4580f85 100644 --- a/builtin/fast-import.c +++ b/builtin/fast-import.c @@ -856,7 +856,7 @@ static void end_packfile(void) struct tag *t; close_pack_windows(pack_data); - finalize_hashfile(pack_file, cur_pack_oid.hash, 0); + finalize_hashfile(pack_file, cur_pack_oid.hash, FSYNC_COMPONENT_PACK, 0); fixup_pack_header_footer(pack_data->pack_fd, pack_data->hash, pack_data->pack_name, object_count, cur_pack_oid.hash, pack_size); diff --git a/builtin/index-pack.c b/builtin/index-pack.c index c23d01de7dc..c32534c13b4 100644 --- a/builtin/index-pack.c +++ b/builtin/index-pack.c @@ -1286,7 +1286,7 @@ static void conclude_pack(int fix_thin_pack, const char *curr_pack, unsigned cha nr_objects - nr_objects_initial); stop_progress_msg(&progress, msg.buf); strbuf_release(&msg); - finalize_hashfile(f, tail_hash, 0); + finalize_hashfile(f, tail_hash, FSYNC_COMPONENT_PACK, 0); hashcpy(read_hash, pack_hash); fixup_pack_header_footer(output_fd, pack_hash, curr_pack, nr_objects, @@ -1508,7 +1508,7 @@ static void final(const char *final_pack_name, const char *curr_pack_name, if (!from_stdin) { close(input_fd); } else { - fsync_or_die(output_fd, curr_pack_name); + fsync_component_or_die(FSYNC_COMPONENT_PACK, output_fd, curr_pack_name); err = close(output_fd); if (err) die_errno(_("error while closing pack file")); diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 857be7826f3..916c55d6ce9 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -1204,11 +1204,13 @@ static void write_pack_file(void) * If so, rewrite it like in fast-import */ if (pack_to_stdout) { - finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_CLOSE); + finalize_hashfile(f, hash, FSYNC_COMPONENT_NONE, + CSUM_HASH_IN_STREAM | CSUM_CLOSE); } else if (nr_written == nr_remaining) { - finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); + finalize_hashfile(f, hash, FSYNC_COMPONENT_PACK, + CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); } else { - int fd = finalize_hashfile(f, hash, 0); + int fd = finalize_hashfile(f, hash, FSYNC_COMPONENT_PACK, 0); fixup_pack_header_footer(fd, hash, pack_tmp_name, nr_written, hash, offset); close(fd); diff --git a/bulk-checkin.c b/bulk-checkin.c index 8785b2ac806..a2cf9dcbc8d 100644 --- a/bulk-checkin.c +++ b/bulk-checkin.c @@ -53,9 +53,10 @@ static void finish_bulk_checkin(struct bulk_checkin_state *state) unlink(state->pack_tmp_name); goto clear_exit; } else if (state->nr_written == 1) { - finalize_hashfile(state->f, hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); + finalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK, + CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); } else { - int fd = finalize_hashfile(state->f, hash, 0); + int fd = finalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK, 0); fixup_pack_header_footer(fd, hash, state->pack_tmp_name, state->nr_written, hash, state->offset); diff --git a/cache.h b/cache.h index 9cd60d94952..d83fbaf2619 100644 --- a/cache.h +++ b/cache.h @@ -985,7 +985,38 @@ void reset_shared_repository(void); extern int read_replace_refs; extern char *git_replace_ref_base; -extern int fsync_object_files; +/* + * These values are used to help identify parts of a repository to fsync. + * FSYNC_COMPONENT_NONE identifies data that will not be a persistent part of the + * repository and so shouldn't be fsynced. + */ +enum fsync_component { + FSYNC_COMPONENT_NONE = 0, + FSYNC_COMPONENT_LOOSE_OBJECT = 1 << 0, + FSYNC_COMPONENT_PACK = 1 << 1, + FSYNC_COMPONENT_PACK_METADATA = 1 << 2, + FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3, +}; + +#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \ + FSYNC_COMPONENT_PACK_METADATA | \ + FSYNC_COMPONENT_COMMIT_GRAPH) + +#define FSYNC_COMPONENTS_OBJECTS (FSYNC_COMPONENT_LOOSE_OBJECT | \ + FSYNC_COMPONENT_PACK | \ + FSYNC_COMPONENT_PACK_METADATA | \ + FSYNC_COMPONENT_COMMIT_GRAPH) + +#define FSYNC_COMPONENTS_ALL (FSYNC_COMPONENT_LOOSE_OBJECT | \ + FSYNC_COMPONENT_PACK | \ + FSYNC_COMPONENT_PACK_METADATA | \ + FSYNC_COMPONENT_COMMIT_GRAPH) + + +/* + * A bitmask indicating which components of the repo should be fsynced. + */ +extern enum fsync_component fsync_components; enum fsync_method { FSYNC_METHOD_FSYNC, @@ -1747,6 +1778,12 @@ int copy_file_with_time(const char *dst, const char *src, int mode); void write_or_die(int fd, const void *buf, size_t count); void fsync_or_die(int fd, const char *); +inline void fsync_component_or_die(enum fsync_component component, int fd, const char *msg) +{ + if (fsync_components & component) + fsync_or_die(fd, msg); +} + ssize_t read_in_full(int fd, void *buf, size_t count); ssize_t write_in_full(int fd, const void *buf, size_t count); ssize_t pread_in_full(int fd, void *buf, size_t count, off_t offset); diff --git a/commit-graph.c b/commit-graph.c index 2706683acfe..c8a5dea4541 100644 --- a/commit-graph.c +++ b/commit-graph.c @@ -1939,7 +1939,8 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx) } close_commit_graph(ctx->r->objects); - finalize_hashfile(f, file_hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC); + finalize_hashfile(f, file_hash, FSYNC_COMPONENT_COMMIT_GRAPH, + CSUM_HASH_IN_STREAM | CSUM_FSYNC); free_chunkfile(cf); if (ctx->split) { diff --git a/config.c b/config.c index c3410b8a868..29c867aab03 100644 --- a/config.c +++ b/config.c @@ -1213,6 +1213,73 @@ static int git_parse_maybe_bool_text(const char *value) return -1; } +static const struct fsync_component_entry { + const char *name; + enum fsync_component component_bits; +} fsync_component_table[] = { + { "loose-object", FSYNC_COMPONENT_LOOSE_OBJECT }, + { "pack", FSYNC_COMPONENT_PACK }, + { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA }, + { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, + { "objects", FSYNC_COMPONENTS_OBJECTS }, + { "default", FSYNC_COMPONENTS_DEFAULT }, + { "all", FSYNC_COMPONENTS_ALL }, +}; + +static enum fsync_component parse_fsync_components(const char *var, const char *string) +{ + enum fsync_component output = 0; + + if (!strcmp(string, "none")) + return output; + + while (string) { + int i; + size_t len; + const char *ep; + int negated = 0; + int found = 0; + + string = string + strspn(string, ", \t\n\r"); + ep = strchrnul(string, ','); + len = ep - string; + + if (*string == '-') { + negated = 1; + string++; + len--; + if (!len) + warning(_("invalid value for variable %s"), var); + } + + if (!len) + break; + + for (i = 0; i < ARRAY_SIZE(fsync_component_table); ++i) { + const struct fsync_component_entry *entry = &fsync_component_table[i]; + + if (strncmp(entry->name, string, len)) + continue; + + found = 1; + if (negated) + output &= ~entry->component_bits; + else + output |= entry->component_bits; + } + + if (!found) { + char *component = xstrndup(string, len); + warning(_("unknown %s value '%s'"), var, component); + free(component); + } + + string = ep; + } + + return output; +} + int git_parse_maybe_bool(const char *value) { int v = git_parse_maybe_bool_text(value); @@ -1490,6 +1557,13 @@ static int git_default_core_config(const char *var, const char *value, void *cb) return 0; } + if (!strcmp(var, "core.fsync")) { + if (!value) + return config_error_nonbool(var); + fsync_components = parse_fsync_components(var, value); + return 0; + } + if (!strcmp(var, "core.fsyncmethod")) { if (!value) return config_error_nonbool(var); @@ -1503,7 +1577,7 @@ static int git_default_core_config(const char *var, const char *value, void *cb) } if (!strcmp(var, "core.fsyncobjectfiles")) { - fsync_object_files = git_config_bool(var, value); + warning(_("core.fsyncobjectfiles is deprecated; use core.fsync instead")); return 0; } diff --git a/csum-file.c b/csum-file.c index 26e8a6df44e..59ef3398ca2 100644 --- a/csum-file.c +++ b/csum-file.c @@ -58,7 +58,8 @@ static void free_hashfile(struct hashfile *f) free(f); } -int finalize_hashfile(struct hashfile *f, unsigned char *result, unsigned int flags) +int finalize_hashfile(struct hashfile *f, unsigned char *result, + enum fsync_component component, unsigned int flags) { int fd; @@ -69,7 +70,7 @@ int finalize_hashfile(struct hashfile *f, unsigned char *result, unsigned int fl if (flags & CSUM_HASH_IN_STREAM) flush(f, f->buffer, the_hash_algo->rawsz); if (flags & CSUM_FSYNC) - fsync_or_die(f->fd, f->name); + fsync_component_or_die(component, f->fd, f->name); if (flags & CSUM_CLOSE) { if (close(f->fd)) die_errno("%s: sha1 file error on close", f->name); diff --git a/csum-file.h b/csum-file.h index 291215b34eb..0d29f528fbc 100644 --- a/csum-file.h +++ b/csum-file.h @@ -1,6 +1,7 @@ #ifndef CSUM_FILE_H #define CSUM_FILE_H +#include "cache.h" #include "hash.h" struct progress; @@ -38,7 +39,7 @@ int hashfile_truncate(struct hashfile *, struct hashfile_checkpoint *); struct hashfile *hashfd(int fd, const char *name); struct hashfile *hashfd_check(const char *name); struct hashfile *hashfd_throughput(int fd, const char *name, struct progress *tp); -int finalize_hashfile(struct hashfile *, unsigned char *, unsigned int); +int finalize_hashfile(struct hashfile *, unsigned char *, enum fsync_component, unsigned int); void hashwrite(struct hashfile *, const void *, unsigned int); void hashflush(struct hashfile *f); void crc32_begin(struct hashfile *); diff --git a/environment.c b/environment.c index f9140e842cf..09905adecf9 100644 --- a/environment.c +++ b/environment.c @@ -42,6 +42,7 @@ const char *git_hooks_path; int zlib_compression_level = Z_BEST_SPEED; int pack_compression_level = Z_DEFAULT_COMPRESSION; enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT; +enum fsync_component fsync_components = FSYNC_COMPONENTS_DEFAULT; size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE; size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT; size_t delta_base_cache_limit = 96 * 1024 * 1024; diff --git a/midx.c b/midx.c index 837b46b2af5..882f91f7d57 100644 --- a/midx.c +++ b/midx.c @@ -1406,7 +1406,8 @@ static int write_midx_internal(const char *object_dir, write_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs); write_chunkfile(cf, &ctx); - finalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM); + finalize_hashfile(f, midx_hash, FSYNC_COMPONENT_PACK_METADATA, + CSUM_FSYNC | CSUM_HASH_IN_STREAM); free_chunkfile(cf); if (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP)) diff --git a/object-file.c b/object-file.c index eb972cdccd2..9d9c4a39e85 100644 --- a/object-file.c +++ b/object-file.c @@ -1809,8 +1809,7 @@ int hash_object_file(const struct git_hash_algo *algo, const void *buf, /* Finalize a file on disk, and close it. */ static void close_loose_object(int fd) { - if (fsync_object_files) - fsync_or_die(fd, "loose object file"); + fsync_component_or_die(FSYNC_COMPONENT_LOOSE_OBJECT, fd, "loose object file"); if (close(fd) != 0) die_errno(_("error when closing loose object file")); } diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c index 9c55c1531e1..c16e43d1669 100644 --- a/pack-bitmap-write.c +++ b/pack-bitmap-write.c @@ -719,7 +719,8 @@ void bitmap_writer_finish(struct pack_idx_entry **index, if (options & BITMAP_OPT_HASH_CACHE) write_hash_cache(f, index, index_nr); - finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); + finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA, + CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); if (adjust_shared_perm(tmp_file.buf)) die_errno("unable to make temporary bitmap file readable"); diff --git a/pack-write.c b/pack-write.c index a5846f3a346..51812cb1299 100644 --- a/pack-write.c +++ b/pack-write.c @@ -159,9 +159,9 @@ const char *write_idx_file(const char *index_name, struct pack_idx_entry **objec } hashwrite(f, sha1, the_hash_algo->rawsz); - finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE | - ((opts->flags & WRITE_IDX_VERIFY) - ? 0 : CSUM_FSYNC)); + finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA, + CSUM_HASH_IN_STREAM | CSUM_CLOSE | + ((opts->flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC)); return index_name; } @@ -281,8 +281,9 @@ const char *write_rev_file_order(const char *rev_name, if (rev_name && adjust_shared_perm(rev_name) < 0) die(_("failed to make %s readable"), rev_name); - finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE | - ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC)); + finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA, + CSUM_HASH_IN_STREAM | CSUM_CLOSE | + ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC)); return rev_name; } @@ -390,7 +391,7 @@ void fixup_pack_header_footer(int pack_fd, the_hash_algo->final_fn(partial_pack_hash, &old_hash_ctx); the_hash_algo->final_fn(new_pack_hash, &new_hash_ctx); write_or_die(pack_fd, new_pack_hash, the_hash_algo->rawsz); - fsync_or_die(pack_fd, pack_name); + fsync_component_or_die(FSYNC_COMPONENT_PACK, pack_fd, pack_name); } char *index_pack_lockfile(int ip_out, int *is_well_formed) diff --git a/read-cache.c b/read-cache.c index f3986596623..f3539681f49 100644 --- a/read-cache.c +++ b/read-cache.c @@ -3060,7 +3060,7 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile, return -1; } - finalize_hashfile(f, istate->oid.hash, CSUM_HASH_IN_STREAM); + finalize_hashfile(f, istate->oid.hash, FSYNC_COMPONENT_NONE, CSUM_HASH_IN_STREAM); if (close_tempfile_gently(tempfile)) { error(_("could not close '%s'"), get_tempfile_path(tempfile)); return -1; -- gitgitgadget ^ permalink raw reply related [flat|nested] 122+ messages in thread
* Re: [PATCH v2 2/3] core.fsync: introduce granular fsync control 2021-12-07 2:46 ` [PATCH v2 2/3] core.fsync: introduce granular fsync control Neeraj Singh via GitGitGadget @ 2021-12-07 11:53 ` Patrick Steinhardt 2021-12-07 20:46 ` Neeraj Singh 2021-12-07 12:29 ` Ævar Arnfjörð Bjarmason 1 sibling, 1 reply; 122+ messages in thread From: Patrick Steinhardt @ 2021-12-07 11:53 UTC (permalink / raw) To: Neeraj Singh via GitGitGadget Cc: git, rsbecker, bagasdotme, newren, avarab, nksingh85, Neeraj K. Singh [-- Attachment #1: Type: text/plain, Size: 11137 bytes --] On Tue, Dec 07, 2021 at 02:46:50AM +0000, Neeraj Singh via GitGitGadget wrote: > From: Neeraj Singh <neerajsi@microsoft.com> [snip] > diff --git a/builtin/index-pack.c b/builtin/index-pack.c > index c23d01de7dc..c32534c13b4 100644 > --- a/builtin/index-pack.c > +++ b/builtin/index-pack.c > @@ -1286,7 +1286,7 @@ static void conclude_pack(int fix_thin_pack, const char *curr_pack, unsigned cha > nr_objects - nr_objects_initial); > stop_progress_msg(&progress, msg.buf); > strbuf_release(&msg); > - finalize_hashfile(f, tail_hash, 0); > + finalize_hashfile(f, tail_hash, FSYNC_COMPONENT_PACK, 0); > hashcpy(read_hash, pack_hash); > fixup_pack_header_footer(output_fd, pack_hash, > curr_pack, nr_objects, > @@ -1508,7 +1508,7 @@ static void final(const char *final_pack_name, const char *curr_pack_name, > if (!from_stdin) { > close(input_fd); > } else { > - fsync_or_die(output_fd, curr_pack_name); > + fsync_component_or_die(FSYNC_COMPONENT_PACK, output_fd, curr_pack_name); > err = close(output_fd); > if (err) > die_errno(_("error while closing pack file")); > diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c > index 857be7826f3..916c55d6ce9 100644 > --- a/builtin/pack-objects.c > +++ b/builtin/pack-objects.c > @@ -1204,11 +1204,13 @@ static void write_pack_file(void) > * If so, rewrite it like in fast-import > */ > if (pack_to_stdout) { > - finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_CLOSE); > + finalize_hashfile(f, hash, FSYNC_COMPONENT_NONE, > + CSUM_HASH_IN_STREAM | CSUM_CLOSE); It doesn't have any effect here given that we don't sync at all when writing to stdout, but I wonder whether we should set up the component correctly regardless of that such that it makes for a less confusing read. [snip] > diff --git a/config.c b/config.c > index c3410b8a868..29c867aab03 100644 > --- a/config.c > +++ b/config.c > @@ -1213,6 +1213,73 @@ static int git_parse_maybe_bool_text(const char *value) > return -1; > } > > +static const struct fsync_component_entry { > + const char *name; > + enum fsync_component component_bits; > +} fsync_component_table[] = { > + { "loose-object", FSYNC_COMPONENT_LOOSE_OBJECT }, > + { "pack", FSYNC_COMPONENT_PACK }, > + { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA }, > + { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, > + { "objects", FSYNC_COMPONENTS_OBJECTS }, > + { "default", FSYNC_COMPONENTS_DEFAULT }, > + { "all", FSYNC_COMPONENTS_ALL }, > +}; > + > +static enum fsync_component parse_fsync_components(const char *var, const char *string) > +{ > + enum fsync_component output = 0; > + > + if (!strcmp(string, "none")) > + return output; > + > + while (string) { > + int i; > + size_t len; > + const char *ep; > + int negated = 0; > + int found = 0; > + > + string = string + strspn(string, ", \t\n\r"); > + ep = strchrnul(string, ','); > + len = ep - string; > + > + if (*string == '-') { > + negated = 1; > + string++; > + len--; > + if (!len) > + warning(_("invalid value for variable %s"), var); > + } > + > + if (!len) > + break; > + > + for (i = 0; i < ARRAY_SIZE(fsync_component_table); ++i) { > + const struct fsync_component_entry *entry = &fsync_component_table[i]; > + > + if (strncmp(entry->name, string, len)) > + continue; > + > + found = 1; > + if (negated) > + output &= ~entry->component_bits; > + else > + output |= entry->component_bits; > + } > + > + if (!found) { > + char *component = xstrndup(string, len); > + warning(_("unknown %s value '%s'"), var, component); > + free(component); > + } > + > + string = ep; > + } > + > + return output; > +} > + > int git_parse_maybe_bool(const char *value) > { > int v = git_parse_maybe_bool_text(value); > @@ -1490,6 +1557,13 @@ static int git_default_core_config(const char *var, const char *value, void *cb) > return 0; > } > > + if (!strcmp(var, "core.fsync")) { > + if (!value) > + return config_error_nonbool(var); > + fsync_components = parse_fsync_components(var, value); > + return 0; > + } > + > if (!strcmp(var, "core.fsyncmethod")) { > if (!value) > return config_error_nonbool(var); > @@ -1503,7 +1577,7 @@ static int git_default_core_config(const char *var, const char *value, void *cb) > } > > if (!strcmp(var, "core.fsyncobjectfiles")) { > - fsync_object_files = git_config_bool(var, value); > + warning(_("core.fsyncobjectfiles is deprecated; use core.fsync instead")); > return 0; > } Shouldn't we continue to support this for now such that users can migrate from the old, deprecated value first before we start to ignore it? Patrick > diff --git a/csum-file.c b/csum-file.c > index 26e8a6df44e..59ef3398ca2 100644 > --- a/csum-file.c > +++ b/csum-file.c > @@ -58,7 +58,8 @@ static void free_hashfile(struct hashfile *f) > free(f); > } > > -int finalize_hashfile(struct hashfile *f, unsigned char *result, unsigned int flags) > +int finalize_hashfile(struct hashfile *f, unsigned char *result, > + enum fsync_component component, unsigned int flags) > { > int fd; > > @@ -69,7 +70,7 @@ int finalize_hashfile(struct hashfile *f, unsigned char *result, unsigned int fl > if (flags & CSUM_HASH_IN_STREAM) > flush(f, f->buffer, the_hash_algo->rawsz); > if (flags & CSUM_FSYNC) > - fsync_or_die(f->fd, f->name); > + fsync_component_or_die(component, f->fd, f->name); > if (flags & CSUM_CLOSE) { > if (close(f->fd)) > die_errno("%s: sha1 file error on close", f->name); > diff --git a/csum-file.h b/csum-file.h > index 291215b34eb..0d29f528fbc 100644 > --- a/csum-file.h > +++ b/csum-file.h > @@ -1,6 +1,7 @@ > #ifndef CSUM_FILE_H > #define CSUM_FILE_H > > +#include "cache.h" > #include "hash.h" > > struct progress; > @@ -38,7 +39,7 @@ int hashfile_truncate(struct hashfile *, struct hashfile_checkpoint *); > struct hashfile *hashfd(int fd, const char *name); > struct hashfile *hashfd_check(const char *name); > struct hashfile *hashfd_throughput(int fd, const char *name, struct progress *tp); > -int finalize_hashfile(struct hashfile *, unsigned char *, unsigned int); > +int finalize_hashfile(struct hashfile *, unsigned char *, enum fsync_component, unsigned int); > void hashwrite(struct hashfile *, const void *, unsigned int); > void hashflush(struct hashfile *f); > void crc32_begin(struct hashfile *); > diff --git a/environment.c b/environment.c > index f9140e842cf..09905adecf9 100644 > --- a/environment.c > +++ b/environment.c > @@ -42,6 +42,7 @@ const char *git_hooks_path; > int zlib_compression_level = Z_BEST_SPEED; > int pack_compression_level = Z_DEFAULT_COMPRESSION; > enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT; > +enum fsync_component fsync_components = FSYNC_COMPONENTS_DEFAULT; > size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE; > size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT; > size_t delta_base_cache_limit = 96 * 1024 * 1024; > diff --git a/midx.c b/midx.c > index 837b46b2af5..882f91f7d57 100644 > --- a/midx.c > +++ b/midx.c > @@ -1406,7 +1406,8 @@ static int write_midx_internal(const char *object_dir, > write_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs); > write_chunkfile(cf, &ctx); > > - finalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM); > + finalize_hashfile(f, midx_hash, FSYNC_COMPONENT_PACK_METADATA, > + CSUM_FSYNC | CSUM_HASH_IN_STREAM); > free_chunkfile(cf); > > if (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP)) > diff --git a/object-file.c b/object-file.c > index eb972cdccd2..9d9c4a39e85 100644 > --- a/object-file.c > +++ b/object-file.c > @@ -1809,8 +1809,7 @@ int hash_object_file(const struct git_hash_algo *algo, const void *buf, > /* Finalize a file on disk, and close it. */ > static void close_loose_object(int fd) > { > - if (fsync_object_files) > - fsync_or_die(fd, "loose object file"); > + fsync_component_or_die(FSYNC_COMPONENT_LOOSE_OBJECT, fd, "loose object file"); > if (close(fd) != 0) > die_errno(_("error when closing loose object file")); > } > diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c > index 9c55c1531e1..c16e43d1669 100644 > --- a/pack-bitmap-write.c > +++ b/pack-bitmap-write.c > @@ -719,7 +719,8 @@ void bitmap_writer_finish(struct pack_idx_entry **index, > if (options & BITMAP_OPT_HASH_CACHE) > write_hash_cache(f, index, index_nr); > > - finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); > + finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA, > + CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); > > if (adjust_shared_perm(tmp_file.buf)) > die_errno("unable to make temporary bitmap file readable"); > diff --git a/pack-write.c b/pack-write.c > index a5846f3a346..51812cb1299 100644 > --- a/pack-write.c > +++ b/pack-write.c > @@ -159,9 +159,9 @@ const char *write_idx_file(const char *index_name, struct pack_idx_entry **objec > } > > hashwrite(f, sha1, the_hash_algo->rawsz); > - finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE | > - ((opts->flags & WRITE_IDX_VERIFY) > - ? 0 : CSUM_FSYNC)); > + finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA, > + CSUM_HASH_IN_STREAM | CSUM_CLOSE | > + ((opts->flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC)); > return index_name; > } > > @@ -281,8 +281,9 @@ const char *write_rev_file_order(const char *rev_name, > if (rev_name && adjust_shared_perm(rev_name) < 0) > die(_("failed to make %s readable"), rev_name); > > - finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE | > - ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC)); > + finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA, > + CSUM_HASH_IN_STREAM | CSUM_CLOSE | > + ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC)); > > return rev_name; > } > @@ -390,7 +391,7 @@ void fixup_pack_header_footer(int pack_fd, > the_hash_algo->final_fn(partial_pack_hash, &old_hash_ctx); > the_hash_algo->final_fn(new_pack_hash, &new_hash_ctx); > write_or_die(pack_fd, new_pack_hash, the_hash_algo->rawsz); > - fsync_or_die(pack_fd, pack_name); > + fsync_component_or_die(FSYNC_COMPONENT_PACK, pack_fd, pack_name); > } > > char *index_pack_lockfile(int ip_out, int *is_well_formed) > diff --git a/read-cache.c b/read-cache.c > index f3986596623..f3539681f49 100644 > --- a/read-cache.c > +++ b/read-cache.c > @@ -3060,7 +3060,7 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile, > return -1; > } > > - finalize_hashfile(f, istate->oid.hash, CSUM_HASH_IN_STREAM); > + finalize_hashfile(f, istate->oid.hash, FSYNC_COMPONENT_NONE, CSUM_HASH_IN_STREAM); > if (close_tempfile_gently(tempfile)) { > error(_("could not close '%s'"), get_tempfile_path(tempfile)); > return -1; > -- > gitgitgadget > [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v2 2/3] core.fsync: introduce granular fsync control 2021-12-07 11:53 ` Patrick Steinhardt @ 2021-12-07 20:46 ` Neeraj Singh 0 siblings, 0 replies; 122+ messages in thread From: Neeraj Singh @ 2021-12-07 20:46 UTC (permalink / raw) To: Patrick Steinhardt Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Ævar Arnfjörð Bjarmason, Neeraj K. Singh On Tue, Dec 7, 2021 at 3:54 AM Patrick Steinhardt <ps@pks.im> wrote: > > On Tue, Dec 07, 2021 at 02:46:50AM +0000, Neeraj Singh via GitGitGadget wrote: > > From: Neeraj Singh <neerajsi@microsoft.com> > [snip] > > diff --git a/builtin/index-pack.c b/builtin/index-pack.c > > index c23d01de7dc..c32534c13b4 100644 > > --- a/builtin/index-pack.c > > +++ b/builtin/index-pack.c > > @@ -1286,7 +1286,7 @@ static void conclude_pack(int fix_thin_pack, const char *curr_pack, unsigned cha > > nr_objects - nr_objects_initial); > > stop_progress_msg(&progress, msg.buf); > > strbuf_release(&msg); > > - finalize_hashfile(f, tail_hash, 0); > > + finalize_hashfile(f, tail_hash, FSYNC_COMPONENT_PACK, 0); > > hashcpy(read_hash, pack_hash); > > fixup_pack_header_footer(output_fd, pack_hash, > > curr_pack, nr_objects, > > @@ -1508,7 +1508,7 @@ static void final(const char *final_pack_name, const char *curr_pack_name, > > if (!from_stdin) { > > close(input_fd); > > } else { > > - fsync_or_die(output_fd, curr_pack_name); > > + fsync_component_or_die(FSYNC_COMPONENT_PACK, output_fd, curr_pack_name); > > err = close(output_fd); > > if (err) > > die_errno(_("error while closing pack file")); > > diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c > > index 857be7826f3..916c55d6ce9 100644 > > --- a/builtin/pack-objects.c > > +++ b/builtin/pack-objects.c > > @@ -1204,11 +1204,13 @@ static void write_pack_file(void) > > * If so, rewrite it like in fast-import > > */ > > if (pack_to_stdout) { > > - finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_CLOSE); > > + finalize_hashfile(f, hash, FSYNC_COMPONENT_NONE, > > + CSUM_HASH_IN_STREAM | CSUM_CLOSE); > > It doesn't have any effect here given that we don't sync at all when > writing to stdout, but I wonder whether we should set up the component > correctly regardless of that such that it makes for a less confusing > read. > If it's not actually a file with some name known to git, is it really a component of the repository? I'd like to leave this one as-is. > [snip] > > diff --git a/config.c b/config.c > > index c3410b8a868..29c867aab03 100644 > > --- a/config.c > > +++ b/config.c > > @@ -1213,6 +1213,73 @@ static int git_parse_maybe_bool_text(const char *value) > > return -1; > > } > > > > +static const struct fsync_component_entry { > > + const char *name; > > + enum fsync_component component_bits; > > +} fsync_component_table[] = { > > + { "loose-object", FSYNC_COMPONENT_LOOSE_OBJECT }, > > + { "pack", FSYNC_COMPONENT_PACK }, > > + { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA }, > > + { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, > > + { "objects", FSYNC_COMPONENTS_OBJECTS }, > > + { "default", FSYNC_COMPONENTS_DEFAULT }, > > + { "all", FSYNC_COMPONENTS_ALL }, > > +}; > > + > > +static enum fsync_component parse_fsync_components(const char *var, const char *string) > > +{ > > + enum fsync_component output = 0; > > + > > + if (!strcmp(string, "none")) > > + return output; > > + > > + while (string) { > > + int i; > > + size_t len; > > + const char *ep; > > + int negated = 0; > > + int found = 0; > > + > > + string = string + strspn(string, ", \t\n\r"); > > + ep = strchrnul(string, ','); > > + len = ep - string; > > + > > + if (*string == '-') { > > + negated = 1; > > + string++; > > + len--; > > + if (!len) > > + warning(_("invalid value for variable %s"), var); > > + } > > + > > + if (!len) > > + break; > > + > > + for (i = 0; i < ARRAY_SIZE(fsync_component_table); ++i) { > > + const struct fsync_component_entry *entry = &fsync_component_table[i]; > > + > > + if (strncmp(entry->name, string, len)) > > + continue; > > + > > + found = 1; > > + if (negated) > > + output &= ~entry->component_bits; > > + else > > + output |= entry->component_bits; > > + } > > + > > + if (!found) { > > + char *component = xstrndup(string, len); > > + warning(_("unknown %s value '%s'"), var, component); > > + free(component); > > + } > > + > > + string = ep; > > + } > > + > > + return output; > > +} > > + > > int git_parse_maybe_bool(const char *value) > > { > > int v = git_parse_maybe_bool_text(value); > > @@ -1490,6 +1557,13 @@ static int git_default_core_config(const char *var, const char *value, void *cb) > > return 0; > > } > > > > + if (!strcmp(var, "core.fsync")) { > > + if (!value) > > + return config_error_nonbool(var); > > + fsync_components = parse_fsync_components(var, value); > > + return 0; > > + } > > + > > if (!strcmp(var, "core.fsyncmethod")) { > > if (!value) > > return config_error_nonbool(var); > > @@ -1503,7 +1577,7 @@ static int git_default_core_config(const char *var, const char *value, void *cb) > > } > > > > if (!strcmp(var, "core.fsyncobjectfiles")) { > > - fsync_object_files = git_config_bool(var, value); > > + warning(_("core.fsyncobjectfiles is deprecated; use core.fsync instead")); > > return 0; > > } > > Shouldn't we continue to support this for now such that users can > migrate from the old, deprecated value first before we start to ignore > it? > > Patrick > That's a good question and one I was hoping to answer through this discussion. I'm guessing that most users do not have this setting explicitly set, and it's largely a non-functional change. Git only behaves differently in the extreme corner case of a system crash after git exits. That's why I believe it's okay to deprecate and remove in one release. If we choose to keep supporting the setting, it would introduce a little complexity in the configuration code, but it should be doable. I think the right semantics would be to ignore core.fsyncobjectfiles if core.fsync is specified with any value. Otherwise, core.fsyncobjectfiles would be equivalent to `default,-loose-object` or `default,loose-object` for `false` and `true` respectively. I'd prefer not to do this if I can get away with it :). Thanks, Neeraj ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v2 2/3] core.fsync: introduce granular fsync control 2021-12-07 2:46 ` [PATCH v2 2/3] core.fsync: introduce granular fsync control Neeraj Singh via GitGitGadget 2021-12-07 11:53 ` Patrick Steinhardt @ 2021-12-07 12:29 ` Ævar Arnfjörð Bjarmason 2021-12-07 21:44 ` Neeraj Singh 1 sibling, 1 reply; 122+ messages in thread From: Ævar Arnfjörð Bjarmason @ 2021-12-07 12:29 UTC (permalink / raw) To: Neeraj Singh via GitGitGadget Cc: git, rsbecker, bagasdotme, newren, nksingh85, ps, Neeraj Singh On Tue, Dec 07 2021, Neeraj Singh via GitGitGadget wrote: > From: Neeraj Singh <neerajsi@microsoft.com> > > This commit introduces the `core.fsync` configuration > knob which can be used to control how components of the > repository are made durable on disk. > > This setting allows future extensibility of the list of > syncable components: > * We issue a warning rather than an error for unrecognized > components, so new configs can be used with old Git versions. Looks good! > * We support negation, so users can choose one of the default > aggregate options and then remove components that they don't > want. The user would then harden any new components added in > a Git version update. I think this config schema makes sense, but just a (I think important) comment on the "how" not "what" of it. It's really much better to define config as: [some] key = value key = value2 Than: [some] key = value,value2 The reason is that "git config" has good support for working with multi-valued stuff, so you can do e.g.: git config --get-all -z some.key And you can easily (re)set such config e.g. with --replace-all etc., but for comma-delimited you (and users) need to do all that work themselves. Similarly instead of: some.key = want-this some.key = -not-this some.key = but-want-this I think it's better to just have two lists, one inclusive another exclusive. E.g. see "log.decorate" and "log.excludeDecoration", "transfer.hideRefs" Which would mean: core.fsync = want-this core.fsyncExcludes = -not-this For some value of "fsyncExcludes", maybe "noFsync"? Anyway, just a suggestion on making this easier for users & the implementation. > This also supports the common request of doing absolutely no > fysncing with the `core.fsync=none` value, which is expected > to make the test suite faster. Let's just use the git_parse_maybe_bool() or git_parse_maybe_bool_text() so we'll accept "false", "off", "no" like most other such config? > Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> > --- > Documentation/config/core.txt | 27 +++++++++---- > builtin/fast-import.c | 2 +- > builtin/index-pack.c | 4 +- > builtin/pack-objects.c | 8 ++-- > bulk-checkin.c | 5 ++- > cache.h | 39 +++++++++++++++++- > commit-graph.c | 3 +- > config.c | 76 ++++++++++++++++++++++++++++++++++- > csum-file.c | 5 ++- > csum-file.h | 3 +- > environment.c | 1 + > midx.c | 3 +- > object-file.c | 3 +- > pack-bitmap-write.c | 3 +- > pack-write.c | 13 +++--- > read-cache.c | 2 +- > 16 files changed, 164 insertions(+), 33 deletions(-) > > diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt > index dbb134f7136..4f1747ec871 100644 > --- a/Documentation/config/core.txt > +++ b/Documentation/config/core.txt > @@ -547,6 +547,25 @@ core.whitespace:: > is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent` > errors. The default tab width is 8. Allowed values are 1 to 63. > > +core.fsync:: > + A comma-separated list of parts of the repository which should be > + hardened via the core.fsyncMethod when created or modified. You can > + disable hardening of any component by prefixing it with a '-'. Later > + items take precedence over earlier ones in the list. For example, > + `core.fsync=all,-pack-metadata` means "harden everything except pack > + metadata." Items that are not hardened may be lost in the event of an > + unclean system shutdown. > ++ > +* `none` disables fsync completely. This must be specified alone. > +* `loose-object` hardens objects added to the repo in loose-object form. > +* `pack` hardens objects added to the repo in packfile form. > +* `pack-metadata` hardens packfile bitmaps and indexes. > +* `commit-graph` hardens the commit graph file. > +* `objects` is an aggregate option that includes `loose-objects`, `pack`, > + `pack-metadata`, and `commit-graph`. > +* `default` is an aggregate option that is equivalent to `objects,-loose-object` > +* `all` is an aggregate option that syncs all individual components above. > + It's probably a *bit* more work to set up, but I wonder if this wouldn't be simpler if we just said (and this is partially going against what I noted above): == BEGIN DOC core.fsync is a multi-value config variable where each item is a pathspec that'll get matched the same way as 'git-ls-files' et al. When we sync pretend that a path like .git/objects/de/adbeef... is relative to the top-level of the git directory. E.g. "objects/de/adbeaf.." or "objects/pack/...". You can then supply a list of wildcards and exclusions to configure syncing. or "false", "off" etc. to turn it off. These are synonymous with: ; same as "false" core.fsync = ":!*" Or: ; same as "true" core.fsync = "*" Or, to selectively sync some things and not others: ;; Sync objects, but not "info" core.fsync = ":!objects/info/**" core.fsync = "objects/**" See gitrepository-layout(5) for details about what sort of paths you might be expected to match. Not all paths listed there will go through this mechanism (e.g. currently objects do, but nothing to do with config does). We can and will match this against "fake paths", e.g. when writing out packs we may match against just the string "objects/pack", we're not going to re-check if every packfile we're writing matches your globs, ditto for loose objects. Be reasonable! This metharism is intended as a shorthand that provides some flexibility when fsyncing, while not forcing git to come up with labels for all paths the git dir, or to support crazyness like "objects/de/adbeef*" More paths may be added or removed in the future, and we make no promises that we won't move things around, so if in doubt use e.g. "true" or a wide pattern match like "objects/**". When in doubt stick to the golden path of examples provided in this documentation. == END DOC It's a tad more complex to set up, but I wonder if that isn't worth it. It nicely gets around any current and future issues of deciding what labels such as "loose-object" etc. to pick, as well as slotting into an existing method of doing exclude/include lists. > diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c > index 857be7826f3..916c55d6ce9 100644 > --- a/builtin/pack-objects.c > +++ b/builtin/pack-objects.c > @@ -1204,11 +1204,13 @@ static void write_pack_file(void) > * If so, rewrite it like in fast-import > */ > if (pack_to_stdout) { > - finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_CLOSE); > + finalize_hashfile(f, hash, FSYNC_COMPONENT_NONE, > + CSUM_HASH_IN_STREAM | CSUM_CLOSE); Not really related to this per-se, but since you're touching the API everything goes through I wonder if callers should just always try to fsync, and we can just catch EROFS and EINVAL in the wrapper if someone tries to flush stdout, or catch the fd at that lower level. Or maybe there's a good reason for this... > [...] > +/* > + * These values are used to help identify parts of a repository to fsync. > + * FSYNC_COMPONENT_NONE identifies data that will not be a persistent part of the > + * repository and so shouldn't be fsynced. > + */ > +enum fsync_component { > + FSYNC_COMPONENT_NONE = 0, I haven't read ahead much but in most other such cases we don't define the "= 0", just start at 1<<0, then check the flags elsewhere... > +static const struct fsync_component_entry { > + const char *name; > + enum fsync_component component_bits; > +} fsync_component_table[] = { > + { "loose-object", FSYNC_COMPONENT_LOOSE_OBJECT }, > + { "pack", FSYNC_COMPONENT_PACK }, > + { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA }, > + { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, > + { "objects", FSYNC_COMPONENTS_OBJECTS }, > + { "default", FSYNC_COMPONENTS_DEFAULT }, > + { "all", FSYNC_COMPONENTS_ALL }, > +}; > + > +static enum fsync_component parse_fsync_components(const char *var, const char *string) > +{ > + enum fsync_component output = 0; > + > + if (!strcmp(string, "none")) > + return output; > + > + while (string) { > + int i; > + size_t len; > + const char *ep; > + int negated = 0; > + int found = 0; > + > + string = string + strspn(string, ", \t\n\r"); Aside from the "use a list" isn't this hardcoding some windows-specific assumptions with \n\r? Maybe not... ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v2 2/3] core.fsync: introduce granular fsync control 2021-12-07 12:29 ` Ævar Arnfjörð Bjarmason @ 2021-12-07 21:44 ` Neeraj Singh 2021-12-08 10:05 ` Ævar Arnfjörð Bjarmason 0 siblings, 1 reply; 122+ messages in thread From: Neeraj Singh @ 2021-12-07 21:44 UTC (permalink / raw) To: Ævar Arnfjörð Bjarmason Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Patrick Steinhardt, Neeraj Singh On Tue, Dec 7, 2021 at 5:01 AM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote: > > > On Tue, Dec 07 2021, Neeraj Singh via GitGitGadget wrote: > > > From: Neeraj Singh <neerajsi@microsoft.com> > > > > This commit introduces the `core.fsync` configuration > > knob which can be used to control how components of the > > repository are made durable on disk. > > > > This setting allows future extensibility of the list of > > syncable components: > > * We issue a warning rather than an error for unrecognized > > components, so new configs can be used with old Git versions. > > Looks good! > > > * We support negation, so users can choose one of the default > > aggregate options and then remove components that they don't > > want. The user would then harden any new components added in > > a Git version update. > > I think this config schema makes sense, but just a (I think important) > comment on the "how" not "what" of it. It's really much better to define > config as: > > [some] > key = value > key = value2 > > Than: > > [some] > key = value,value2 > > The reason is that "git config" has good support for working with > multi-valued stuff, so you can do e.g.: > > git config --get-all -z some.key > > And you can easily (re)set such config e.g. with --replace-all etc., but > for comma-delimited you (and users) need to do all that work themselves. > > Similarly instead of: > > some.key = want-this > some.key = -not-this > some.key = but-want-this > > I think it's better to just have two lists, one inclusive another > exclusive. E.g. see "log.decorate" and "log.excludeDecoration", > "transfer.hideRefs" > > Which would mean: > > core.fsync = want-this > core.fsyncExcludes = -not-this > > For some value of "fsyncExcludes", maybe "noFsync"? Anyway, just a > suggestion on making this easier for users & the implementation. > Maybe there's some way to handle this I'm unaware of, but a disadvantage of your multi-valued config proposal is that it's harder, for example, for a per-repo config store to reasonably override a per-user config store. With the configuration scheme as-is, I can have a per-user setting like `core.fsync=all` which covers my typical repos, but then have a maintainer repo with a private setting of `core.fsync=none` to speed up cases where I'm mostly working with other people's changes that are backed up in email or server-side repos. The latter setting conveniently overrides the former setting in all aspects. Also, with the core.fsync and core.fsyncExcludes, how would you spell "don't sync anything"? Would you still have the aggregate options.? > > This also supports the common request of doing absolutely no > > fysncing with the `core.fsync=none` value, which is expected > > to make the test suite faster. > > Let's just use the git_parse_maybe_bool() or git_parse_maybe_bool_text() > so we'll accept "false", "off", "no" like most other such config? Junio's previous feedback when discussing batch mode [1] was to offer less flexibility when parsing new values of these configuration options. I agree with his statement that "making machine-readable tokens be spelled in different ways is a 'disease'." I'd like to leave this as-is so that the documentation can clearly state the exact set of allowable values. [1] https://lore.kernel.org/git/xmqqr1dqzyl7.fsf@gitster.g/ > > > Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> > > --- > > Documentation/config/core.txt | 27 +++++++++---- > > builtin/fast-import.c | 2 +- > > builtin/index-pack.c | 4 +- > > builtin/pack-objects.c | 8 ++-- > > bulk-checkin.c | 5 ++- > > cache.h | 39 +++++++++++++++++- > > commit-graph.c | 3 +- > > config.c | 76 ++++++++++++++++++++++++++++++++++- > > csum-file.c | 5 ++- > > csum-file.h | 3 +- > > environment.c | 1 + > > midx.c | 3 +- > > object-file.c | 3 +- > > pack-bitmap-write.c | 3 +- > > pack-write.c | 13 +++--- > > read-cache.c | 2 +- > > 16 files changed, 164 insertions(+), 33 deletions(-) > > > > diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt > > index dbb134f7136..4f1747ec871 100644 > > --- a/Documentation/config/core.txt > > +++ b/Documentation/config/core.txt > > @@ -547,6 +547,25 @@ core.whitespace:: > > is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent` > > errors. The default tab width is 8. Allowed values are 1 to 63. > > > > +core.fsync:: > > + A comma-separated list of parts of the repository which should be > > + hardened via the core.fsyncMethod when created or modified. You can > > + disable hardening of any component by prefixing it with a '-'. Later > > + items take precedence over earlier ones in the list. For example, > > + `core.fsync=all,-pack-metadata` means "harden everything except pack > > + metadata." Items that are not hardened may be lost in the event of an > > + unclean system shutdown. > > ++ > > +* `none` disables fsync completely. This must be specified alone. > > +* `loose-object` hardens objects added to the repo in loose-object form. > > +* `pack` hardens objects added to the repo in packfile form. > > +* `pack-metadata` hardens packfile bitmaps and indexes. > > +* `commit-graph` hardens the commit graph file. > > +* `objects` is an aggregate option that includes `loose-objects`, `pack`, > > + `pack-metadata`, and `commit-graph`. > > +* `default` is an aggregate option that is equivalent to `objects,-loose-object` > > +* `all` is an aggregate option that syncs all individual components above. > > + > > It's probably a *bit* more work to set up, but I wonder if this wouldn't > be simpler if we just said (and this is partially going against what I > noted above): > > == BEGIN DOC > > core.fsync is a multi-value config variable where each item is a > pathspec that'll get matched the same way as 'git-ls-files' et al. > > When we sync pretend that a path like .git/objects/de/adbeef... is > relative to the top-level of the git > directory. E.g. "objects/de/adbeaf.." or "objects/pack/...". > > You can then supply a list of wildcards and exclusions to configure > syncing. or "false", "off" etc. to turn it off. These are synonymous > with: > > ; same as "false" > core.fsync = ":!*" > > Or: > > ; same as "true" > core.fsync = "*" > > Or, to selectively sync some things and not others: > > ;; Sync objects, but not "info" > core.fsync = ":!objects/info/**" > core.fsync = "objects/**" > > See gitrepository-layout(5) for details about what sort of paths you > might be expected to match. Not all paths listed there will go through > this mechanism (e.g. currently objects do, but nothing to do with config > does). > > We can and will match this against "fake paths", e.g. when writing out > packs we may match against just the string "objects/pack", we're not > going to re-check if every packfile we're writing matches your globs, > ditto for loose objects. Be reasonable! > > This metharism is intended as a shorthand that provides some flexibility > when fsyncing, while not forcing git to come up with labels for all > paths the git dir, or to support crazyness like "objects/de/adbeef*" > > More paths may be added or removed in the future, and we make no > promises that we won't move things around, so if in doubt use > e.g. "true" or a wide pattern match like "objects/**". When in doubt > stick to the golden path of examples provided in this documentation. > > == END DOC > > > It's a tad more complex to set up, but I wonder if that isn't worth > it. It nicely gets around any current and future issues of deciding what > labels such as "loose-object" etc. to pick, as well as slotting into an > existing method of doing exclude/include lists. > I think this proposal is a lot of complexity to avoid coming up with a new name for syncable things as they are added to Git. A path based mechanism makes it hard to document for the (advanced) user what the full set of things is and how it might change from release to release. I think the current core.fsync scheme is a bit easier to understand, query, and extend. > > diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c > > index 857be7826f3..916c55d6ce9 100644 > > --- a/builtin/pack-objects.c > > +++ b/builtin/pack-objects.c > > @@ -1204,11 +1204,13 @@ static void write_pack_file(void) > > * If so, rewrite it like in fast-import > > */ > > if (pack_to_stdout) { > > - finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_CLOSE); > > + finalize_hashfile(f, hash, FSYNC_COMPONENT_NONE, > > + CSUM_HASH_IN_STREAM | CSUM_CLOSE); > > Not really related to this per-se, but since you're touching the API > everything goes through I wonder if callers should just always try to > fsync, and we can just catch EROFS and EINVAL in the wrapper if someone > tries to flush stdout, or catch the fd at that lower level. > > Or maybe there's a good reason for this... It's platform dependent, but I'd expect fsync would do something for pipes or stdout redirected to a file. In these cases we really don't want to fsync since we have no idea what we're talking to and we're potentially worsening performance for probably no benefit. > > [...] > > +/* > > + * These values are used to help identify parts of a repository to fsync. > > + * FSYNC_COMPONENT_NONE identifies data that will not be a persistent part of the > > + * repository and so shouldn't be fsynced. > > + */ > > +enum fsync_component { > > + FSYNC_COMPONENT_NONE = 0, > > I haven't read ahead much but in most other such cases we don't define > the "= 0", just start at 1<<0, then check the flags elsewhere... > > > +static const struct fsync_component_entry { > > + const char *name; > > + enum fsync_component component_bits; > > +} fsync_component_table[] = { > > + { "loose-object", FSYNC_COMPONENT_LOOSE_OBJECT }, > > + { "pack", FSYNC_COMPONENT_PACK }, > > + { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA }, > > + { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, > > + { "objects", FSYNC_COMPONENTS_OBJECTS }, > > + { "default", FSYNC_COMPONENTS_DEFAULT }, > > + { "all", FSYNC_COMPONENTS_ALL }, > > +}; > > + > > +static enum fsync_component parse_fsync_components(const char *var, const char *string) > > +{ > > + enum fsync_component output = 0; > > + > > + if (!strcmp(string, "none")) > > + return output; > > + > > + while (string) { > > + int i; > > + size_t len; > > + const char *ep; > > + int negated = 0; > > + int found = 0; > > + > > + string = string + strspn(string, ", \t\n\r"); > > Aside from the "use a list" isn't this hardcoding some windows-specific > assumptions with \n\r? Maybe not... I shamelessly stole this code from parse_whitespace_rule. I thought about making a helper to be called by both functions, but the amount of state going into and out of the wrapper via arguments was substantial and seemed to negate the benefit of deduplication. Thanks for the review, Neeraj ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v2 2/3] core.fsync: introduce granular fsync control 2021-12-07 21:44 ` Neeraj Singh @ 2021-12-08 10:05 ` Ævar Arnfjörð Bjarmason 2021-12-09 0:14 ` Neeraj Singh 0 siblings, 1 reply; 122+ messages in thread From: Ævar Arnfjörð Bjarmason @ 2021-12-08 10:05 UTC (permalink / raw) To: Neeraj Singh Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Patrick Steinhardt, Neeraj Singh On Tue, Dec 07 2021, Neeraj Singh wrote: > On Tue, Dec 7, 2021 at 5:01 AM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote: >> >> >> On Tue, Dec 07 2021, Neeraj Singh via GitGitGadget wrote: >> >> > From: Neeraj Singh <neerajsi@microsoft.com> >> > >> > This commit introduces the `core.fsync` configuration >> > knob which can be used to control how components of the >> > repository are made durable on disk. >> > >> > This setting allows future extensibility of the list of >> > syncable components: >> > * We issue a warning rather than an error for unrecognized >> > components, so new configs can be used with old Git versions. >> >> Looks good! >> >> > * We support negation, so users can choose one of the default >> > aggregate options and then remove components that they don't >> > want. The user would then harden any new components added in >> > a Git version update. >> >> I think this config schema makes sense, but just a (I think important) >> comment on the "how" not "what" of it. It's really much better to define >> config as: >> >> [some] >> key = value >> key = value2 >> >> Than: >> >> [some] >> key = value,value2 >> >> The reason is that "git config" has good support for working with >> multi-valued stuff, so you can do e.g.: >> >> git config --get-all -z some.key >> >> And you can easily (re)set such config e.g. with --replace-all etc., but >> for comma-delimited you (and users) need to do all that work themselves. >> >> Similarly instead of: >> >> some.key = want-this >> some.key = -not-this >> some.key = but-want-this >> >> I think it's better to just have two lists, one inclusive another >> exclusive. E.g. see "log.decorate" and "log.excludeDecoration", >> "transfer.hideRefs" >> >> Which would mean: >> >> core.fsync = want-this >> core.fsyncExcludes = -not-this >> >> For some value of "fsyncExcludes", maybe "noFsync"? Anyway, just a >> suggestion on making this easier for users & the implementation. >> > > Maybe there's some way to handle this I'm unaware of, but a > disadvantage of your multi-valued config proposal is that it's harder, > for example, for a per-repo config store to reasonably override a > per-user config store. With the configuration scheme as-is, I can > have a per-user setting like `core.fsync=all` which covers my typical > repos, but then have a maintainer repo with a private setting of > `core.fsync=none` to speed up cases where I'm mostly working with > other people's changes that are backed up in email or server-side > repos. The latter setting conveniently overrides the former setting > in all aspects. Even if you turn just your comma-delimited proposal into a list proposal can't we just say that the last one wins? Then it won't matter what cmae before, you'd specify "core.fsync=none" in your local .git/config. But this is also a general issue with a bunch of things in git's config space. I'd rather see us use the list-based values and just come up with some general way to reset them that works for all keys, rather than regretting having comma-delimited values that'll be harder to work with & parse, which will be a legacy wart if/when we come up with a way to say "reset all previous settings". > Also, with the core.fsync and core.fsyncExcludes, how would you spell > "don't sync anything"? Would you still have the aggregate options.? core.fsyncExcludes = * I.e. the same as the core.fsync=none above, anyway I still like the wildcard thing below a bit more... >> > This also supports the common request of doing absolutely no >> > fysncing with the `core.fsync=none` value, which is expected >> > to make the test suite faster. >> >> Let's just use the git_parse_maybe_bool() or git_parse_maybe_bool_text() >> so we'll accept "false", "off", "no" like most other such config? > > Junio's previous feedback when discussing batch mode [1] was to offer > less flexibility when parsing new values of these configuration > options. I agree with his statement that "making machine-readable > tokens be spelled in different ways is a 'disease'." I'd like to > leave this as-is so that the documentation can clearly state the exact > set of allowable values. > > [1] https://lore.kernel.org/git/xmqqr1dqzyl7.fsf@gitster.g/ I think he's talking about batch, Batch, BATCH, bAtCh etc. there. But the "maybe bool" is a stanard pattern we use. I don't think we'd call one of these 0, off, no or false etc. to avoid confusion, so then you can use git_parse_maybe_...() >> >> > Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> >> > --- >> > Documentation/config/core.txt | 27 +++++++++---- >> > builtin/fast-import.c | 2 +- >> > builtin/index-pack.c | 4 +- >> > builtin/pack-objects.c | 8 ++-- >> > bulk-checkin.c | 5 ++- >> > cache.h | 39 +++++++++++++++++- >> > commit-graph.c | 3 +- >> > config.c | 76 ++++++++++++++++++++++++++++++++++- >> > csum-file.c | 5 ++- >> > csum-file.h | 3 +- >> > environment.c | 1 + >> > midx.c | 3 +- >> > object-file.c | 3 +- >> > pack-bitmap-write.c | 3 +- >> > pack-write.c | 13 +++--- >> > read-cache.c | 2 +- >> > 16 files changed, 164 insertions(+), 33 deletions(-) >> > >> > diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt >> > index dbb134f7136..4f1747ec871 100644 >> > --- a/Documentation/config/core.txt >> > +++ b/Documentation/config/core.txt >> > @@ -547,6 +547,25 @@ core.whitespace:: >> > is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent` >> > errors. The default tab width is 8. Allowed values are 1 to 63. >> > >> > +core.fsync:: >> > + A comma-separated list of parts of the repository which should be >> > + hardened via the core.fsyncMethod when created or modified. You can >> > + disable hardening of any component by prefixing it with a '-'. Later >> > + items take precedence over earlier ones in the list. For example, >> > + `core.fsync=all,-pack-metadata` means "harden everything except pack >> > + metadata." Items that are not hardened may be lost in the event of an >> > + unclean system shutdown. >> > ++ >> > +* `none` disables fsync completely. This must be specified alone. >> > +* `loose-object` hardens objects added to the repo in loose-object form. >> > +* `pack` hardens objects added to the repo in packfile form. >> > +* `pack-metadata` hardens packfile bitmaps and indexes. >> > +* `commit-graph` hardens the commit graph file. >> > +* `objects` is an aggregate option that includes `loose-objects`, `pack`, >> > + `pack-metadata`, and `commit-graph`. >> > +* `default` is an aggregate option that is equivalent to `objects,-loose-object` >> > +* `all` is an aggregate option that syncs all individual components above. >> > + >> >> It's probably a *bit* more work to set up, but I wonder if this wouldn't >> be simpler if we just said (and this is partially going against what I >> noted above): >> >> == BEGIN DOC >> >> core.fsync is a multi-value config variable where each item is a >> pathspec that'll get matched the same way as 'git-ls-files' et al. >> >> When we sync pretend that a path like .git/objects/de/adbeef... is >> relative to the top-level of the git >> directory. E.g. "objects/de/adbeaf.." or "objects/pack/...". >> >> You can then supply a list of wildcards and exclusions to configure >> syncing. or "false", "off" etc. to turn it off. These are synonymous >> with: >> >> ; same as "false" >> core.fsync = ":!*" >> >> Or: >> >> ; same as "true" >> core.fsync = "*" >> >> Or, to selectively sync some things and not others: >> >> ;; Sync objects, but not "info" >> core.fsync = ":!objects/info/**" >> core.fsync = "objects/**" >> >> See gitrepository-layout(5) for details about what sort of paths you >> might be expected to match. Not all paths listed there will go through >> this mechanism (e.g. currently objects do, but nothing to do with config >> does). >> >> We can and will match this against "fake paths", e.g. when writing out >> packs we may match against just the string "objects/pack", we're not >> going to re-check if every packfile we're writing matches your globs, >> ditto for loose objects. Be reasonable! >> >> This metharism is intended as a shorthand that provides some flexibility >> when fsyncing, while not forcing git to come up with labels for all >> paths the git dir, or to support crazyness like "objects/de/adbeef*" >> >> More paths may be added or removed in the future, and we make no >> promises that we won't move things around, so if in doubt use >> e.g. "true" or a wide pattern match like "objects/**". When in doubt >> stick to the golden path of examples provided in this documentation. >> >> == END DOC >> >> >> It's a tad more complex to set up, but I wonder if that isn't worth >> it. It nicely gets around any current and future issues of deciding what >> labels such as "loose-object" etc. to pick, as well as slotting into an >> existing method of doing exclude/include lists. >> > > I think this proposal is a lot of complexity to avoid coming up with a > new name for syncable things as they are added to Git. A path based > mechanism makes it hard to document for the (advanced) user what the > full set of things is and how it might change from release to release. > I think the current core.fsync scheme is a bit easier to understand, > query, and extend. We document it in gitrepository-layout(5). Yeah it has some disadvantages, but one advantage is that you could make the composability easy. I.e. if last exclude wins then a setting of: core.fsync = ":!*" core.fsync = "objects/**" Would reset all previous matches & only match objects/**. >> > diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c >> > index 857be7826f3..916c55d6ce9 100644 >> > --- a/builtin/pack-objects.c >> > +++ b/builtin/pack-objects.c >> > @@ -1204,11 +1204,13 @@ static void write_pack_file(void) >> > * If so, rewrite it like in fast-import >> > */ >> > if (pack_to_stdout) { >> > - finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_CLOSE); >> > + finalize_hashfile(f, hash, FSYNC_COMPONENT_NONE, >> > + CSUM_HASH_IN_STREAM | CSUM_CLOSE); >> >> Not really related to this per-se, but since you're touching the API >> everything goes through I wonder if callers should just always try to >> fsync, and we can just catch EROFS and EINVAL in the wrapper if someone >> tries to flush stdout, or catch the fd at that lower level. >> >> Or maybe there's a good reason for this... > > It's platform dependent, but I'd expect fsync would do something for > pipes or stdout redirected to a file. In these cases we really don't > want to fsync since we have no idea what we're talking to and we're > potentially worsening performance for probably no benefit. Yeah maybe we should just leave it be. I'd think the C library returning EINVAL would be a trivial performance cost though. It just seemed odd to hardcode assumptions about what can and can't be synced when the POSIX defined function will also tell us that. Anyway... >> > [...] >> > +/* >> > + * These values are used to help identify parts of a repository to fsync. >> > + * FSYNC_COMPONENT_NONE identifies data that will not be a persistent part of the >> > + * repository and so shouldn't be fsynced. >> > + */ >> > +enum fsync_component { >> > + FSYNC_COMPONENT_NONE = 0, >> >> I haven't read ahead much but in most other such cases we don't define >> the "= 0", just start at 1<<0, then check the flags elsewhere... >> >> > +static const struct fsync_component_entry { >> > + const char *name; >> > + enum fsync_component component_bits; >> > +} fsync_component_table[] = { >> > + { "loose-object", FSYNC_COMPONENT_LOOSE_OBJECT }, >> > + { "pack", FSYNC_COMPONENT_PACK }, >> > + { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA }, >> > + { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, >> > + { "objects", FSYNC_COMPONENTS_OBJECTS }, >> > + { "default", FSYNC_COMPONENTS_DEFAULT }, >> > + { "all", FSYNC_COMPONENTS_ALL }, >> > +}; >> > + >> > +static enum fsync_component parse_fsync_components(const char *var, const char *string) >> > +{ >> > + enum fsync_component output = 0; >> > + >> > + if (!strcmp(string, "none")) >> > + return output; >> > + >> > + while (string) { >> > + int i; >> > + size_t len; >> > + const char *ep; >> > + int negated = 0; >> > + int found = 0; >> > + >> > + string = string + strspn(string, ", \t\n\r"); >> >> Aside from the "use a list" isn't this hardcoding some windows-specific >> assumptions with \n\r? Maybe not... > > I shamelessly stole this code from parse_whitespace_rule. I thought > about making a helper to be called by both functions, but the amount > of state going into and out of the wrapper via arguments was > substantial and seemed to negate the benefit of deduplication. FWIW string_list_split() is easier to work with in those cases, or at least I think so... ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v2 2/3] core.fsync: introduce granular fsync control 2021-12-08 10:05 ` Ævar Arnfjörð Bjarmason @ 2021-12-09 0:14 ` Neeraj Singh 2021-12-09 0:44 ` Junio C Hamano 2021-12-09 4:08 ` Ævar Arnfjörð Bjarmason 0 siblings, 2 replies; 122+ messages in thread From: Neeraj Singh @ 2021-12-09 0:14 UTC (permalink / raw) To: Ævar Arnfjörð Bjarmason Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Patrick Steinhardt, Neeraj Singh On Wed, Dec 8, 2021 at 2:17 AM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote: > > > On Tue, Dec 07 2021, Neeraj Singh wrote: > > > On Tue, Dec 7, 2021 at 5:01 AM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote: > >> > >> > >> On Tue, Dec 07 2021, Neeraj Singh via GitGitGadget wrote: > >> > >> > From: Neeraj Singh <neerajsi@microsoft.com> > >> > > >> > This commit introduces the `core.fsync` configuration > >> > knob which can be used to control how components of the > >> > repository are made durable on disk. > >> > > >> > This setting allows future extensibility of the list of > >> > syncable components: > >> > * We issue a warning rather than an error for unrecognized > >> > components, so new configs can be used with old Git versions. > >> > >> Looks good! > >> > >> > * We support negation, so users can choose one of the default > >> > aggregate options and then remove components that they don't > >> > want. The user would then harden any new components added in > >> > a Git version update. > >> > >> I think this config schema makes sense, but just a (I think important) > >> comment on the "how" not "what" of it. It's really much better to define > >> config as: > >> > >> [some] > >> key = value > >> key = value2 > >> > >> Than: > >> > >> [some] > >> key = value,value2 > >> > >> The reason is that "git config" has good support for working with > >> multi-valued stuff, so you can do e.g.: > >> > >> git config --get-all -z some.key > >> > >> And you can easily (re)set such config e.g. with --replace-all etc., but > >> for comma-delimited you (and users) need to do all that work themselves. > >> > >> Similarly instead of: > >> > >> some.key = want-this > >> some.key = -not-this > >> some.key = but-want-this > >> > >> I think it's better to just have two lists, one inclusive another > >> exclusive. E.g. see "log.decorate" and "log.excludeDecoration", > >> "transfer.hideRefs" > >> > >> Which would mean: > >> > >> core.fsync = want-this > >> core.fsyncExcludes = -not-this > >> > >> For some value of "fsyncExcludes", maybe "noFsync"? Anyway, just a > >> suggestion on making this easier for users & the implementation. > >> > > > > Maybe there's some way to handle this I'm unaware of, but a > > disadvantage of your multi-valued config proposal is that it's harder, > > for example, for a per-repo config store to reasonably override a > > per-user config store. With the configuration scheme as-is, I can > > have a per-user setting like `core.fsync=all` which covers my typical > > repos, but then have a maintainer repo with a private setting of > > `core.fsync=none` to speed up cases where I'm mostly working with > > other people's changes that are backed up in email or server-side > > repos. The latter setting conveniently overrides the former setting > > in all aspects. > > Even if you turn just your comma-delimited proposal into a list proposal > can't we just say that the last one wins? Then it won't matter what cmae > before, you'd specify "core.fsync=none" in your local .git/config. > > But this is also a general issue with a bunch of things in git's config > space. I'd rather see us use the list-based values and just come up with > some general way to reset them that works for all keys, rather than > regretting having comma-delimited values that'll be harder to work with > & parse, which will be a legacy wart if/when we come up with a way to > say "reset all previous settings". > > > Also, with the core.fsync and core.fsyncExcludes, how would you spell > > "don't sync anything"? Would you still have the aggregate options.? > > core.fsyncExcludes = * > > I.e. the same as the core.fsync=none above, anyway I still like the > wildcard thing below a bit more... I'm not going to take this feedback unless there are additional votes from the Git community in this direction. I make the claim that single-valued comma-separated config lists are easier to work with in the existing Git infrastructure. We already use essentially the same parsing code for the core.whitespace variable and users are used to this syntax there. There are several other comma-separated lists in the config space, so this construct has precedence and will be with Git for some time. Also, fsync configurations aren't composable like some other configurations may be. It makes sense to have a holistic singular fsync configuration, which is best represented by a single variable. > >> > This also supports the common request of doing absolutely no > >> > fysncing with the `core.fsync=none` value, which is expected > >> > to make the test suite faster. > >> > >> Let's just use the git_parse_maybe_bool() or git_parse_maybe_bool_text() > >> so we'll accept "false", "off", "no" like most other such config? > > > > Junio's previous feedback when discussing batch mode [1] was to offer > > less flexibility when parsing new values of these configuration > > options. I agree with his statement that "making machine-readable > > tokens be spelled in different ways is a 'disease'." I'd like to > > leave this as-is so that the documentation can clearly state the exact > > set of allowable values. > > > > [1] https://lore.kernel.org/git/xmqqr1dqzyl7.fsf@gitster.g/ > > I think he's talking about batch, Batch, BATCH, bAtCh etc. there. But > the "maybe bool" is a stanard pattern we use. > > I don't think we'd call one of these 0, off, no or false etc. to avoid > confusion, so then you can use git_parse_maybe_...() I don't see the advantage of having multiple ways of specifying "none". The user can read the doc and know exactly what to write. If they write something unallowable, they get a clear warning and they can read the doc again to figure out what to write. This isn't a boolean options at all, so why should we entertain bool-like ways of spelling it? > >> > >> > Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> > >> > --- > >> > Documentation/config/core.txt | 27 +++++++++---- > >> > builtin/fast-import.c | 2 +- > >> > builtin/index-pack.c | 4 +- > >> > builtin/pack-objects.c | 8 ++-- > >> > bulk-checkin.c | 5 ++- > >> > cache.h | 39 +++++++++++++++++- > >> > commit-graph.c | 3 +- > >> > config.c | 76 ++++++++++++++++++++++++++++++++++- > >> > csum-file.c | 5 ++- > >> > csum-file.h | 3 +- > >> > environment.c | 1 + > >> > midx.c | 3 +- > >> > object-file.c | 3 +- > >> > pack-bitmap-write.c | 3 +- > >> > pack-write.c | 13 +++--- > >> > read-cache.c | 2 +- > >> > 16 files changed, 164 insertions(+), 33 deletions(-) > >> > > >> > diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt > >> > index dbb134f7136..4f1747ec871 100644 > >> > --- a/Documentation/config/core.txt > >> > +++ b/Documentation/config/core.txt > >> > @@ -547,6 +547,25 @@ core.whitespace:: > >> > is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent` > >> > errors. The default tab width is 8. Allowed values are 1 to 63. > >> > > >> > +core.fsync:: > >> > + A comma-separated list of parts of the repository which should be > >> > + hardened via the core.fsyncMethod when created or modified. You can > >> > + disable hardening of any component by prefixing it with a '-'. Later > >> > + items take precedence over earlier ones in the list. For example, > >> > + `core.fsync=all,-pack-metadata` means "harden everything except pack > >> > + metadata." Items that are not hardened may be lost in the event of an > >> > + unclean system shutdown. > >> > ++ > >> > +* `none` disables fsync completely. This must be specified alone. > >> > +* `loose-object` hardens objects added to the repo in loose-object form. > >> > +* `pack` hardens objects added to the repo in packfile form. > >> > +* `pack-metadata` hardens packfile bitmaps and indexes. > >> > +* `commit-graph` hardens the commit graph file. > >> > +* `objects` is an aggregate option that includes `loose-objects`, `pack`, > >> > + `pack-metadata`, and `commit-graph`. > >> > +* `default` is an aggregate option that is equivalent to `objects,-loose-object` > >> > +* `all` is an aggregate option that syncs all individual components above. > >> > + > >> > >> It's probably a *bit* more work to set up, but I wonder if this wouldn't > >> be simpler if we just said (and this is partially going against what I > >> noted above): > >> > >> == BEGIN DOC > >> > >> core.fsync is a multi-value config variable where each item is a > >> pathspec that'll get matched the same way as 'git-ls-files' et al. > >> > >> When we sync pretend that a path like .git/objects/de/adbeef... is > >> relative to the top-level of the git > >> directory. E.g. "objects/de/adbeaf.." or "objects/pack/...". > >> > >> You can then supply a list of wildcards and exclusions to configure > >> syncing. or "false", "off" etc. to turn it off. These are synonymous > >> with: > >> > >> ; same as "false" > >> core.fsync = ":!*" > >> > >> Or: > >> > >> ; same as "true" > >> core.fsync = "*" > >> > >> Or, to selectively sync some things and not others: > >> > >> ;; Sync objects, but not "info" > >> core.fsync = ":!objects/info/**" > >> core.fsync = "objects/**" > >> > >> See gitrepository-layout(5) for details about what sort of paths you > >> might be expected to match. Not all paths listed there will go through > >> this mechanism (e.g. currently objects do, but nothing to do with config > >> does). > >> > >> We can and will match this against "fake paths", e.g. when writing out > >> packs we may match against just the string "objects/pack", we're not > >> going to re-check if every packfile we're writing matches your globs, > >> ditto for loose objects. Be reasonable! > >> > >> This metharism is intended as a shorthand that provides some flexibility > >> when fsyncing, while not forcing git to come up with labels for all > >> paths the git dir, or to support crazyness like "objects/de/adbeef*" > >> > >> More paths may be added or removed in the future, and we make no > >> promises that we won't move things around, so if in doubt use > >> e.g. "true" or a wide pattern match like "objects/**". When in doubt > >> stick to the golden path of examples provided in this documentation. > >> > >> == END DOC > >> > >> > >> It's a tad more complex to set up, but I wonder if that isn't worth > >> it. It nicely gets around any current and future issues of deciding what > >> labels such as "loose-object" etc. to pick, as well as slotting into an > >> existing method of doing exclude/include lists. > >> > > > > I think this proposal is a lot of complexity to avoid coming up with a > > new name for syncable things as they are added to Git. A path based > > mechanism makes it hard to document for the (advanced) user what the > > full set of things is and how it might change from release to release. > > I think the current core.fsync scheme is a bit easier to understand, > > query, and extend. > > We document it in gitrepository-layout(5). Yeah it has some > disadvantages, but one advantage is that you could make the > composability easy. I.e. if last exclude wins then a setting of: > > core.fsync = ":!*" > core.fsync = "objects/**" > > Would reset all previous matches & only match objects/**. > The value of changing this is predicated on taking your previous multi-valued config proposal, which I'm still not at all convinced about. The schema in the current (v1-v2) version of the patch already includes an example of extending the list of syncable things, and Patrick Steinhardt made it clear that he feels comfortable adding 'refs' to the same schema in a future change. I'll also emphasize that we're talking about a non-functional, relatively corner-case behavioral configuration. These values don't change how git's interface behaves except when the system crashes during a git command or shortly after one completes. While you may not personally love the proposed configuration interface, I'd want your view on some questions: 1. Is it easy for the (advanced) user to set a configuration? 2. Is it easy for the (advanced) user to see what was configured? 3. Is it easy for the Git community to build on this as we want to add things to the list of things to sync? a) Is there a good best practice configuration so that people can avoid losing integrity for new stuff that they are intending to sync. b) If someone has a custom configuration, can that custom configuration do something reasonable as they upgrade versions of Git? ** In response to this question, I might see some value in adding a 'derived-metadata' aggregate that can be disabled so that a custom configuration can exclude those as they change version to version. c) Is it too much maintenance overhead to consider how to present this configuration knob for any new hashfile or other datafile in the git repo? 4. Is there a good path forward to change the default syncable set, both in git-for-windows and in Git for other platforms? > >> > diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c > >> > index 857be7826f3..916c55d6ce9 100644 > >> > --- a/builtin/pack-objects.c > >> > +++ b/builtin/pack-objects.c > >> > @@ -1204,11 +1204,13 @@ static void write_pack_file(void) > >> > * If so, rewrite it like in fast-import > >> > */ > >> > if (pack_to_stdout) { > >> > - finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_CLOSE); > >> > + finalize_hashfile(f, hash, FSYNC_COMPONENT_NONE, > >> > + CSUM_HASH_IN_STREAM | CSUM_CLOSE); > >> > >> Not really related to this per-se, but since you're touching the API > >> everything goes through I wonder if callers should just always try to > >> fsync, and we can just catch EROFS and EINVAL in the wrapper if someone > >> tries to flush stdout, or catch the fd at that lower level. > >> > >> Or maybe there's a good reason for this... > > > > It's platform dependent, but I'd expect fsync would do something for > > pipes or stdout redirected to a file. In these cases we really don't > > want to fsync since we have no idea what we're talking to and we're > > potentially worsening performance for probably no benefit. > > Yeah maybe we should just leave it be. > > I'd think the C library returning EINVAL would be a trivial performance > cost though. > > It just seemed odd to hardcode assumptions about what can and can't be > synced when the POSIX defined function will also tell us that. > Redirecting stdout to a file seems like a common usage for this command. That would definitely be fsyncable, but Git has no idea what its proper category is since there's no way to know the purpose or lifetime of the packfile. I'm going to leave this be, because I'd posit that "can it be fsynced?" is not the same as "should it be fsynced?". The latter question can't be answered for stdout. > > >> > [...] > >> > +/* > >> > + * These values are used to help identify parts of a repository to fsync. > >> > + * FSYNC_COMPONENT_NONE identifies data that will not be a persistent part of the > >> > + * repository and so shouldn't be fsynced. > >> > + */ > >> > +enum fsync_component { > >> > + FSYNC_COMPONENT_NONE = 0, > >> > >> I haven't read ahead much but in most other such cases we don't define > >> the "= 0", just start at 1<<0, then check the flags elsewhere... > >> > >> > +static const struct fsync_component_entry { > >> > + const char *name; > >> > + enum fsync_component component_bits; > >> > +} fsync_component_table[] = { > >> > + { "loose-object", FSYNC_COMPONENT_LOOSE_OBJECT }, > >> > + { "pack", FSYNC_COMPONENT_PACK }, > >> > + { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA }, > >> > + { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, > >> > + { "objects", FSYNC_COMPONENTS_OBJECTS }, > >> > + { "default", FSYNC_COMPONENTS_DEFAULT }, > >> > + { "all", FSYNC_COMPONENTS_ALL }, > >> > +}; > >> > + > >> > +static enum fsync_component parse_fsync_components(const char *var, const char *string) > >> > +{ > >> > + enum fsync_component output = 0; > >> > + > >> > + if (!strcmp(string, "none")) > >> > + return output; > >> > + > >> > + while (string) { > >> > + int i; > >> > + size_t len; > >> > + const char *ep; > >> > + int negated = 0; > >> > + int found = 0; > >> > + > >> > + string = string + strspn(string, ", \t\n\r"); > >> > >> Aside from the "use a list" isn't this hardcoding some windows-specific > >> assumptions with \n\r? Maybe not... > > > > I shamelessly stole this code from parse_whitespace_rule. I thought > > about making a helper to be called by both functions, but the amount > > of state going into and out of the wrapper via arguments was > > substantial and seemed to negate the benefit of deduplication. > > FWIW string_list_split() is easier to work with in those cases, or at > least I think so... This code runs at startup for a variable that may be present on some installations. The nice property of the current patch's code is that it's already a well-tested pattern that doesn't do any allocations as it's working, unlike string_list_split(). I hope you know that I appreciate your review feedback, even though I'm pushing back on most of it so far this round. I'll be sending v3 to the list soon after giving it another look over. Thanks, Neeraj ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v2 2/3] core.fsync: introduce granular fsync control 2021-12-09 0:14 ` Neeraj Singh @ 2021-12-09 0:44 ` Junio C Hamano 2021-12-09 4:08 ` Ævar Arnfjörð Bjarmason 1 sibling, 0 replies; 122+ messages in thread From: Junio C Hamano @ 2021-12-09 0:44 UTC (permalink / raw) To: Neeraj Singh Cc: Ævar Arnfjörð Bjarmason, Neeraj Singh via GitGitGadget, Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Patrick Steinhardt, Neeraj Singh Neeraj Singh <nksingh85@gmail.com> writes: > I'm not going to take this feedback unless there are additional votes > from the Git community in this direction. I make the claim that > single-valued comma-separated config lists are easier to work with in > the existing Git infrastructure. We already use essentially the same > parsing code for the core.whitespace variable and users are used to > this syntax there. There are several other comma-separated lists in > the config space, so this construct has precedence and will be with > Git for some time. Also, fsync configurations aren't composable like > some other configurations may be. It makes sense to have a holistic > singular fsync configuration, which is best represented by a single > variable. I haven't caught up with the discussion in this thread, even though I have been meaning to think about it for some time---I just haven't got around to it (sorry). So I'll stop at giving a general guidance and leave the decision if it applies to this particular discussion to readers. As the inventor of core.whitespace "list of values and its parser, I am biased, but I would say that it works well for simple things that do not need too much overriding. The other side of the coin is that it can become very awkward going forward if we use it to things that have more complex needs than answering a simple question like "what whitespace errors should be checked?". More specifically, core.whitespace is pecuriar in a few ways. * It does follow the usual "the last one wins" rule, but in a strange way. Notice the "unsigned rule = WS_DEFAULT_RULE" assignment at the beginning of ws.c::parse_whitespace_rule()? For each configuration "core.whitespace=<list>" we encounter, we start from the default, discarding everything we saw so far, and tweak that default value with tokens found on the list. * You cannot do "The system config gives one set of values, which you tweak with the personal config, which is further tweaked with the repository config" as the consequence. This is blessing and is curse at the same time, as it makes inspection simpler when things go wrong (you only need to check whta the last one does), but it is harder to share the common things in more common file. * Its design relies on the choices being strings chosen from a fixed vocabulary to allow you to say "the value of the configuration variable is a list of tokens separated by a comma" and "the default has X bit set, but we disable that with -X". For a configuration variable whose value is an arbitrary string or a number, obviously that approach would not work. If the need of the topic is simple enough that the above limitation core.whitespace does not pose a problem going forward, it would be fine, but we may regret the choice we make today if that is not the case. Thanks. ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v2 2/3] core.fsync: introduce granular fsync control 2021-12-09 0:14 ` Neeraj Singh 2021-12-09 0:44 ` Junio C Hamano @ 2021-12-09 4:08 ` Ævar Arnfjörð Bjarmason 2021-12-09 6:18 ` Neeraj Singh 1 sibling, 1 reply; 122+ messages in thread From: Ævar Arnfjörð Bjarmason @ 2021-12-09 4:08 UTC (permalink / raw) To: Neeraj Singh Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Patrick Steinhardt, Neeraj Singh On Wed, Dec 08 2021, Neeraj Singh wrote: > On Wed, Dec 8, 2021 at 2:17 AM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote: >> >> >> On Tue, Dec 07 2021, Neeraj Singh wrote: >> >> > On Tue, Dec 7, 2021 at 5:01 AM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote: >> >> >> >> >> >> On Tue, Dec 07 2021, Neeraj Singh via GitGitGadget wrote: >> >> >> >> > From: Neeraj Singh <neerajsi@microsoft.com> >> >> > >> >> > This commit introduces the `core.fsync` configuration >> >> > knob which can be used to control how components of the >> >> > repository are made durable on disk. >> >> > >> >> > This setting allows future extensibility of the list of >> >> > syncable components: >> >> > * We issue a warning rather than an error for unrecognized >> >> > components, so new configs can be used with old Git versions. >> >> >> >> Looks good! >> >> >> >> > * We support negation, so users can choose one of the default >> >> > aggregate options and then remove components that they don't >> >> > want. The user would then harden any new components added in >> >> > a Git version update. >> >> >> >> I think this config schema makes sense, but just a (I think important) >> >> comment on the "how" not "what" of it. It's really much better to define >> >> config as: >> >> >> >> [some] >> >> key = value >> >> key = value2 >> >> >> >> Than: >> >> >> >> [some] >> >> key = value,value2 >> >> >> >> The reason is that "git config" has good support for working with >> >> multi-valued stuff, so you can do e.g.: >> >> >> >> git config --get-all -z some.key >> >> >> >> And you can easily (re)set such config e.g. with --replace-all etc., but >> >> for comma-delimited you (and users) need to do all that work themselves. >> >> >> >> Similarly instead of: >> >> >> >> some.key = want-this >> >> some.key = -not-this >> >> some.key = but-want-this >> >> >> >> I think it's better to just have two lists, one inclusive another >> >> exclusive. E.g. see "log.decorate" and "log.excludeDecoration", >> >> "transfer.hideRefs" >> >> >> >> Which would mean: >> >> >> >> core.fsync = want-this >> >> core.fsyncExcludes = -not-this >> >> >> >> For some value of "fsyncExcludes", maybe "noFsync"? Anyway, just a >> >> suggestion on making this easier for users & the implementation. >> >> >> > >> > Maybe there's some way to handle this I'm unaware of, but a >> > disadvantage of your multi-valued config proposal is that it's harder, >> > for example, for a per-repo config store to reasonably override a >> > per-user config store. With the configuration scheme as-is, I can >> > have a per-user setting like `core.fsync=all` which covers my typical >> > repos, but then have a maintainer repo with a private setting of >> > `core.fsync=none` to speed up cases where I'm mostly working with >> > other people's changes that are backed up in email or server-side >> > repos. The latter setting conveniently overrides the former setting >> > in all aspects. >> >> Even if you turn just your comma-delimited proposal into a list proposal >> can't we just say that the last one wins? Then it won't matter what cmae >> before, you'd specify "core.fsync=none" in your local .git/config. >> >> But this is also a general issue with a bunch of things in git's config >> space. I'd rather see us use the list-based values and just come up with >> some general way to reset them that works for all keys, rather than >> regretting having comma-delimited values that'll be harder to work with >> & parse, which will be a legacy wart if/when we come up with a way to >> say "reset all previous settings". >> >> > Also, with the core.fsync and core.fsyncExcludes, how would you spell >> > "don't sync anything"? Would you still have the aggregate options.? >> >> core.fsyncExcludes = * >> >> I.e. the same as the core.fsync=none above, anyway I still like the >> wildcard thing below a bit more... > > I'm not going to take this feedback unless there are additional votes > from the Git community in this direction. I make the claim that > single-valued comma-separated config lists are easier to work with in > the existing Git infrastructure. Easier in what sense? I showed examples of how "git-config" trivially works with multi-valued config, but for comma-delimited you'll need to write your own shellscript boilerplate around simple things like adding values, removing existing ones etc. I.e. you can use --add, --unset, --unset-all, --get, --get-all etc. > parsing code for the core.whitespace variable and users are used to > this syntax there. There are several other comma-separated lists in > the config space, so this construct has precedence and will be with > Git for some time. That's not really an argument either way for why we'd pick X over Y for something new. We've got some comma-delimited, some multi-value (I'm fairly sure we have more multi-value). > Also, fsync configurations aren't composable like > some other configurations may be. It makes sense to have a holistic > singular fsync configuration, which is best represented by a single > variable. What's a "variable" here? We call these "keys", you can have a single-value key like user.name that you get with --get, or a multi-value key like say branch.<name>.merge or push.pushOption that you'd get with --get-all. I think you may be referring to either not wanting these to be "inherited" (which is not really a think we do for anything else in config), or lacking the ability to "reset". For the latter if that's absolutely needed we could just use the same trick as "diff.wsErrorHighlight" uses of making an empty value "reset" the list, and you'd get better "git config" support for editing it. >> >> > This also supports the common request of doing absolutely no >> >> > fysncing with the `core.fsync=none` value, which is expected >> >> > to make the test suite faster. >> >> >> >> Let's just use the git_parse_maybe_bool() or git_parse_maybe_bool_text() >> >> so we'll accept "false", "off", "no" like most other such config? >> > >> > Junio's previous feedback when discussing batch mode [1] was to offer >> > less flexibility when parsing new values of these configuration >> > options. I agree with his statement that "making machine-readable >> > tokens be spelled in different ways is a 'disease'." I'd like to >> > leave this as-is so that the documentation can clearly state the exact >> > set of allowable values. >> > >> > [1] https://lore.kernel.org/git/xmqqr1dqzyl7.fsf@gitster.g/ >> >> I think he's talking about batch, Batch, BATCH, bAtCh etc. there. But >> the "maybe bool" is a stanard pattern we use. >> >> I don't think we'd call one of these 0, off, no or false etc. to avoid >> confusion, so then you can use git_parse_maybe_...() > > I don't see the advantage of having multiple ways of specifying > "none". The user can read the doc and know exactly what to write. If > they write something unallowable, they get a clear warning and they > can read the doc again to figure out what to write. This isn't a > boolean options at all, so why should we entertain bool-like ways of > spelling it? It's not boolean, it's multi-value and one of the values includes a true or false boolean value. You just spell it "none". I think both this and your comment above suggest that you think there's no point in this because you haven't interacted with/used "git config" as a command line or API mechanism, but have just hand-crafted config files. That's fair enough, but there's a lot of tooling that benefits from the latter. E.g.: $ git -c core.fsync=off config --type=bool core.fsync false $ git -c core.fsync=blah config --type=bool core.fsync fatal: bad boolean config value 'blah' for 'core.fsync' Here we can get 'git config' to normalize what you call 'none', and you can tell via exit codes/normalization if it's "false". But if you invent a new term for "false" you can't do that as easily. We have various historical keys that take odd exceptions to that, e.g. "never", but unless we have a good reason to let's not invent more exceptions. >> >> > Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> >> >> > --- >> >> > Documentation/config/core.txt | 27 +++++++++---- >> >> > builtin/fast-import.c | 2 +- >> >> > builtin/index-pack.c | 4 +- >> >> > builtin/pack-objects.c | 8 ++-- >> >> > bulk-checkin.c | 5 ++- >> >> > cache.h | 39 +++++++++++++++++- >> >> > commit-graph.c | 3 +- >> >> > config.c | 76 ++++++++++++++++++++++++++++++++++- >> >> > csum-file.c | 5 ++- >> >> > csum-file.h | 3 +- >> >> > environment.c | 1 + >> >> > midx.c | 3 +- >> >> > object-file.c | 3 +- >> >> > pack-bitmap-write.c | 3 +- >> >> > pack-write.c | 13 +++--- >> >> > read-cache.c | 2 +- >> >> > 16 files changed, 164 insertions(+), 33 deletions(-) >> >> > >> >> > diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt >> >> > index dbb134f7136..4f1747ec871 100644 >> >> > --- a/Documentation/config/core.txt >> >> > +++ b/Documentation/config/core.txt >> >> > @@ -547,6 +547,25 @@ core.whitespace:: >> >> > is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent` >> >> > errors. The default tab width is 8. Allowed values are 1 to 63. >> >> > >> >> > +core.fsync:: >> >> > + A comma-separated list of parts of the repository which should be >> >> > + hardened via the core.fsyncMethod when created or modified. You can >> >> > + disable hardening of any component by prefixing it with a '-'. Later >> >> > + items take precedence over earlier ones in the list. For example, >> >> > + `core.fsync=all,-pack-metadata` means "harden everything except pack >> >> > + metadata." Items that are not hardened may be lost in the event of an >> >> > + unclean system shutdown. >> >> > ++ >> >> > +* `none` disables fsync completely. This must be specified alone. >> >> > +* `loose-object` hardens objects added to the repo in loose-object form. >> >> > +* `pack` hardens objects added to the repo in packfile form. >> >> > +* `pack-metadata` hardens packfile bitmaps and indexes. >> >> > +* `commit-graph` hardens the commit graph file. >> >> > +* `objects` is an aggregate option that includes `loose-objects`, `pack`, >> >> > + `pack-metadata`, and `commit-graph`. >> >> > +* `default` is an aggregate option that is equivalent to `objects,-loose-object` >> >> > +* `all` is an aggregate option that syncs all individual components above. >> >> > + >> >> >> >> It's probably a *bit* more work to set up, but I wonder if this wouldn't >> >> be simpler if we just said (and this is partially going against what I >> >> noted above): >> >> >> >> == BEGIN DOC >> >> >> >> core.fsync is a multi-value config variable where each item is a >> >> pathspec that'll get matched the same way as 'git-ls-files' et al. >> >> >> >> When we sync pretend that a path like .git/objects/de/adbeef... is >> >> relative to the top-level of the git >> >> directory. E.g. "objects/de/adbeaf.." or "objects/pack/...". >> >> >> >> You can then supply a list of wildcards and exclusions to configure >> >> syncing. or "false", "off" etc. to turn it off. These are synonymous >> >> with: >> >> >> >> ; same as "false" >> >> core.fsync = ":!*" >> >> >> >> Or: >> >> >> >> ; same as "true" >> >> core.fsync = "*" >> >> >> >> Or, to selectively sync some things and not others: >> >> >> >> ;; Sync objects, but not "info" >> >> core.fsync = ":!objects/info/**" >> >> core.fsync = "objects/**" >> >> >> >> See gitrepository-layout(5) for details about what sort of paths you >> >> might be expected to match. Not all paths listed there will go through >> >> this mechanism (e.g. currently objects do, but nothing to do with config >> >> does). >> >> >> >> We can and will match this against "fake paths", e.g. when writing out >> >> packs we may match against just the string "objects/pack", we're not >> >> going to re-check if every packfile we're writing matches your globs, >> >> ditto for loose objects. Be reasonable! >> >> >> >> This metharism is intended as a shorthand that provides some flexibility >> >> when fsyncing, while not forcing git to come up with labels for all >> >> paths the git dir, or to support crazyness like "objects/de/adbeef*" >> >> >> >> More paths may be added or removed in the future, and we make no >> >> promises that we won't move things around, so if in doubt use >> >> e.g. "true" or a wide pattern match like "objects/**". When in doubt >> >> stick to the golden path of examples provided in this documentation. >> >> >> >> == END DOC >> >> >> >> >> >> It's a tad more complex to set up, but I wonder if that isn't worth >> >> it. It nicely gets around any current and future issues of deciding what >> >> labels such as "loose-object" etc. to pick, as well as slotting into an >> >> existing method of doing exclude/include lists. >> >> >> > >> > I think this proposal is a lot of complexity to avoid coming up with a >> > new name for syncable things as they are added to Git. A path based >> > mechanism makes it hard to document for the (advanced) user what the >> > full set of things is and how it might change from release to release. >> > I think the current core.fsync scheme is a bit easier to understand, >> > query, and extend. >> >> We document it in gitrepository-layout(5). Yeah it has some >> disadvantages, but one advantage is that you could make the >> composability easy. I.e. if last exclude wins then a setting of: >> >> core.fsync = ":!*" >> core.fsync = "objects/**" >> >> Would reset all previous matches & only match objects/**. >> > > The value of changing this is predicated on taking your previous > multi-valued config proposal, which I'm still not at all convinced > about. They're orthagonal. I.e. you get benefits from multi-value with or without this globbing mechanism. In any case, I don't feel strongly about/am really advocating this globbing mechanism. I just wondered if it wouldn't make things simpler since it would sidestep the need to create any sort of categories for subsets of gitrepository-layout(5), but maybe not... > The schema in the current (v1-v2) version of the patch already > includes an example of extending the list of syncable things, and > Patrick Steinhardt made it clear that he feels comfortable adding > 'refs' to the same schema in a future change. > > I'll also emphasize that we're talking about a non-functional, > relatively corner-case behavioral configuration. These values don't > change how git's interface behaves except when the system crashes > during a git command or shortly after one completes. That may be something some OS's promise, but it's not something fsync() or POSIX promises. I.e. you might write a ref, but unless you fsync and the relevant dir entries another process might not see the update, crash or not. That's an aside from these config design questions, and I think most/(all?) OS's anyone cares about these days tend to make that implicit promise as part of their VFS behavior, but we should probably design such an interface to fsync() with such pedantic portability in mind. > While you may not personally love the proposed configuration > interface, I'd want your view on some questions: > 1. Is it easy for the (advanced) user to set a configuration? > 2. Is it easy for the (advanced) user to see what was configured? > 3. Is it easy for the Git community to build on this as we want to add > things to the list of things to sync? > a) Is there a good best practice configuration so that people can > avoid losing integrity for new stuff that they are intending to sync. > b) If someone has a custom configuration, can that custom > configuration do something reasonable as they upgrade versions of Git? > ** In response to this question, I might see some value > in adding a 'derived-metadata' aggregate that can be disabled so that > a custom configuration can exclude those as they change version to > version. > c) Is it too much maintenance overhead to consider how to present > this configuration knob for any new hashfile or other datafile in the > git repo? > 4. Is there a good path forward to change the default syncable set, > both in git-for-windows and in Git for other platforms? I'm not really sure this globbing this is a good idea, as noted above just a suggestion etc. As noted there it just gets you out of the business of re-defining gitrepository-layout(5), and assuming too much in advance about certain use-cases. E.g. even "refs" might be too broad for some. I don't tend to be I/O limited, but I could see how someone who would be would care about refs/heads but not refs/remotes, or want to exclude logs/* but not the refs updates themselves etc. >> >> > diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c >> >> > index 857be7826f3..916c55d6ce9 100644 >> >> > --- a/builtin/pack-objects.c >> >> > +++ b/builtin/pack-objects.c >> >> > @@ -1204,11 +1204,13 @@ static void write_pack_file(void) >> >> > * If so, rewrite it like in fast-import >> >> > */ >> >> > if (pack_to_stdout) { >> >> > - finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_CLOSE); >> >> > + finalize_hashfile(f, hash, FSYNC_COMPONENT_NONE, >> >> > + CSUM_HASH_IN_STREAM | CSUM_CLOSE); >> >> >> >> Not really related to this per-se, but since you're touching the API >> >> everything goes through I wonder if callers should just always try to >> >> fsync, and we can just catch EROFS and EINVAL in the wrapper if someone >> >> tries to flush stdout, or catch the fd at that lower level. >> >> >> >> Or maybe there's a good reason for this... >> > >> > It's platform dependent, but I'd expect fsync would do something for >> > pipes or stdout redirected to a file. In these cases we really don't >> > want to fsync since we have no idea what we're talking to and we're >> > potentially worsening performance for probably no benefit. >> >> Yeah maybe we should just leave it be. >> >> I'd think the C library returning EINVAL would be a trivial performance >> cost though. >> >> It just seemed odd to hardcode assumptions about what can and can't be >> synced when the POSIX defined function will also tell us that. >> > > Redirecting stdout to a file seems like a common usage for this > command. That would definitely be fsyncable, but Git has no idea what > its proper category is since there's no way to know the purpose or > lifetime of the packfile. I'm going to leave this be, because I'd > posit that "can it be fsynced?" is not the same as "should it be > fsynced?". The latter question can't be answered for stdout. As noted this was just an aside, and I don't even know if any OS would do anything meaningful with an fsync() of such a FD anyway. I just don't see why we wouldn't say: 1. We're syncing this category of thing 2. Try it 3. If fsync returns "can't fsync that sort of thing" move on As opposed to trying to shortcut #3 by doing the detection ourselves. I.e. maybe there was a good reason, but it seemed to be some easy potential win for more simplification, since you were re-doing and simplifying some of the interface anyway... >> >> >> > [...] >> >> > +/* >> >> > + * These values are used to help identify parts of a repository to fsync. >> >> > + * FSYNC_COMPONENT_NONE identifies data that will not be a persistent part of the >> >> > + * repository and so shouldn't be fsynced. >> >> > + */ >> >> > +enum fsync_component { >> >> > + FSYNC_COMPONENT_NONE = 0, >> >> >> >> I haven't read ahead much but in most other such cases we don't define >> >> the "= 0", just start at 1<<0, then check the flags elsewhere... >> >> >> >> > +static const struct fsync_component_entry { >> >> > + const char *name; >> >> > + enum fsync_component component_bits; >> >> > +} fsync_component_table[] = { >> >> > + { "loose-object", FSYNC_COMPONENT_LOOSE_OBJECT }, >> >> > + { "pack", FSYNC_COMPONENT_PACK }, >> >> > + { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA }, >> >> > + { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, >> >> > + { "objects", FSYNC_COMPONENTS_OBJECTS }, >> >> > + { "default", FSYNC_COMPONENTS_DEFAULT }, >> >> > + { "all", FSYNC_COMPONENTS_ALL }, >> >> > +}; >> >> > + >> >> > +static enum fsync_component parse_fsync_components(const char *var, const char *string) >> >> > +{ >> >> > + enum fsync_component output = 0; >> >> > + >> >> > + if (!strcmp(string, "none")) >> >> > + return output; >> >> > + >> >> > + while (string) { >> >> > + int i; >> >> > + size_t len; >> >> > + const char *ep; >> >> > + int negated = 0; >> >> > + int found = 0; >> >> > + >> >> > + string = string + strspn(string, ", \t\n\r"); >> >> >> >> Aside from the "use a list" isn't this hardcoding some windows-specific >> >> assumptions with \n\r? Maybe not... >> > >> > I shamelessly stole this code from parse_whitespace_rule. I thought >> > about making a helper to be called by both functions, but the amount >> > of state going into and out of the wrapper via arguments was >> > substantial and seemed to negate the benefit of deduplication. >> >> FWIW string_list_split() is easier to work with in those cases, or at >> least I think so... > > This code runs at startup for a variable that may be present on some > installations. The nice property of the current patch's code is that > it's already a well-tested pattern that doesn't do any allocations as > it's working, unlike string_list_split(). Multi-value config would also get you fewer allocations :) Anyway, I mainly meant to point out that for stuff like this it's fine to optimize it for ease rather than micro-optimize allocations. Those really aren't a bottleneck on this scale. Even in that case there's string_list_split_in_place(), which can be a bit nicer than manual C-string fiddling. > I hope you know that I appreciate your review feedback, even though > I'm pushing back on most of it so far this round. I'll be sending v3 > to the list soon after giving it another look over. Sure, no worries. Just hoping to help. If you go for something different etc. that's fine. Just hoping to bridge the gap in some knowledge / offer potentially interesting suggestions (some of which may be dead ends, like the config glob thing...). ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v2 2/3] core.fsync: introduce granular fsync control 2021-12-09 4:08 ` Ævar Arnfjörð Bjarmason @ 2021-12-09 6:18 ` Neeraj Singh 2022-01-18 23:50 ` Neeraj Singh 2022-01-19 14:52 ` Ævar Arnfjörð Bjarmason 0 siblings, 2 replies; 122+ messages in thread From: Neeraj Singh @ 2021-12-09 6:18 UTC (permalink / raw) To: Ævar Arnfjörð Bjarmason Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Patrick Steinhardt, Neeraj Singh On Wed, Dec 8, 2021 at 8:46 PM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote: > > > On Wed, Dec 08 2021, Neeraj Singh wrote: > > > On Wed, Dec 8, 2021 at 2:17 AM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote: > >> > >> > >> On Tue, Dec 07 2021, Neeraj Singh wrote: > >> > >> > On Tue, Dec 7, 2021 at 5:01 AM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote: > >> >> > >> >> > >> >> On Tue, Dec 07 2021, Neeraj Singh via GitGitGadget wrote: > >> >> > >> >> > From: Neeraj Singh <neerajsi@microsoft.com> > >> >> > > >> >> > This commit introduces the `core.fsync` configuration > >> >> > knob which can be used to control how components of the > >> >> > repository are made durable on disk. > >> >> > > >> >> > This setting allows future extensibility of the list of > >> >> > syncable components: > >> >> > * We issue a warning rather than an error for unrecognized > >> >> > components, so new configs can be used with old Git versions. > >> >> > >> >> Looks good! > >> >> > >> >> > * We support negation, so users can choose one of the default > >> >> > aggregate options and then remove components that they don't > >> >> > want. The user would then harden any new components added in > >> >> > a Git version update. > >> >> > >> >> I think this config schema makes sense, but just a (I think important) > >> >> comment on the "how" not "what" of it. It's really much better to define > >> >> config as: > >> >> > >> >> [some] > >> >> key = value > >> >> key = value2 > >> >> > >> >> Than: > >> >> > >> >> [some] > >> >> key = value,value2 > >> >> > >> >> The reason is that "git config" has good support for working with > >> >> multi-valued stuff, so you can do e.g.: > >> >> > >> >> git config --get-all -z some.key > >> >> > >> >> And you can easily (re)set such config e.g. with --replace-all etc., but > >> >> for comma-delimited you (and users) need to do all that work themselves. > >> >> > >> >> Similarly instead of: > >> >> > >> >> some.key = want-this > >> >> some.key = -not-this > >> >> some.key = but-want-this > >> >> > >> >> I think it's better to just have two lists, one inclusive another > >> >> exclusive. E.g. see "log.decorate" and "log.excludeDecoration", > >> >> "transfer.hideRefs" > >> >> > >> >> Which would mean: > >> >> > >> >> core.fsync = want-this > >> >> core.fsyncExcludes = -not-this > >> >> > >> >> For some value of "fsyncExcludes", maybe "noFsync"? Anyway, just a > >> >> suggestion on making this easier for users & the implementation. > >> >> > >> > > >> > Maybe there's some way to handle this I'm unaware of, but a > >> > disadvantage of your multi-valued config proposal is that it's harder, > >> > for example, for a per-repo config store to reasonably override a > >> > per-user config store. With the configuration scheme as-is, I can > >> > have a per-user setting like `core.fsync=all` which covers my typical > >> > repos, but then have a maintainer repo with a private setting of > >> > `core.fsync=none` to speed up cases where I'm mostly working with > >> > other people's changes that are backed up in email or server-side > >> > repos. The latter setting conveniently overrides the former setting > >> > in all aspects. > >> > >> Even if you turn just your comma-delimited proposal into a list proposal > >> can't we just say that the last one wins? Then it won't matter what cmae > >> before, you'd specify "core.fsync=none" in your local .git/config. > >> > >> But this is also a general issue with a bunch of things in git's config > >> space. I'd rather see us use the list-based values and just come up with > >> some general way to reset them that works for all keys, rather than > >> regretting having comma-delimited values that'll be harder to work with > >> & parse, which will be a legacy wart if/when we come up with a way to > >> say "reset all previous settings". > >> > >> > Also, with the core.fsync and core.fsyncExcludes, how would you spell > >> > "don't sync anything"? Would you still have the aggregate options.? > >> > >> core.fsyncExcludes = * > >> > >> I.e. the same as the core.fsync=none above, anyway I still like the > >> wildcard thing below a bit more... > > > > I'm not going to take this feedback unless there are additional votes > > from the Git community in this direction. I make the claim that > > single-valued comma-separated config lists are easier to work with in > > the existing Git infrastructure. > > Easier in what sense? I showed examples of how "git-config" trivially > works with multi-valued config, but for comma-delimited you'll need to > write your own shellscript boilerplate around simple things like adding > values, removing existing ones etc. > > I.e. you can use --add, --unset, --unset-all, --get, --get-all etc. > I see what you're saying for cases where someone would want to set a core.fsync setting that's derived from the user's current config. But I'm guessing that the dominant use case is someone setting a new fsync configuration that achieves some atomic goal with respect to a given repository. Like "this is a throwaway, so sync nothing" or "this is really important, so sync all objects and refs and the index". > > parsing code for the core.whitespace variable and users are used to > > this syntax there. There are several other comma-separated lists in > > the config space, so this construct has precedence and will be with > > Git for some time. > > That's not really an argument either way for why we'd pick X over Y for > something new. We've got some comma-delimited, some multi-value (I'm > fairly sure we have more multi-value). > My main point here is that there's precedent for patch's current exact schema for a config in the same core config leaf of the Documentation. It seems from your comments that we'd have to invent and document some new convention for "reset" of a multi-valued config. So you're suggesting that I solve an extra set of problems to get this change in. Just want to remind you that my personal itch to scratch is to get the underlying mechanism in so that git-for-windows can set its default setting to batch mode. I'm not expecting many users to actually configure this setting to any non-default value. > > Also, fsync configurations aren't composable like > > some other configurations may be. It makes sense to have a holistic > > singular fsync configuration, which is best represented by a single > > variable. > > What's a "variable" here? We call these "keys", you can have a > single-value key like user.name that you get with --get, or a > multi-value key like say branch.<name>.merge or push.pushOption that > you'd get with --get-all. Yeah, I meant "key". I conflated the config key with the underlying global variable in git. > I think you may be referring to either not wanting these to be > "inherited" (which is not really a think we do for anything else in > config), or lacking the ability to "reset". > > For the latter if that's absolutely needed we could just use the same > trick as "diff.wsErrorHighlight" uses of making an empty value "reset" > the list, and you'd get better "git config" support for editing it. > My reading of the code is that diff.wsErrorHighlight is a comma separated list and not a multi-valued config. Actually I haven't yet found an existing multi-valued config (not sure how to grep for it). > >> >> > This also supports the common request of doing absolutely no > >> >> > fysncing with the `core.fsync=none` value, which is expected > >> >> > to make the test suite faster. > >> >> > >> >> Let's just use the git_parse_maybe_bool() or git_parse_maybe_bool_text() > >> >> so we'll accept "false", "off", "no" like most other such config? > >> > > >> > Junio's previous feedback when discussing batch mode [1] was to offer > >> > less flexibility when parsing new values of these configuration > >> > options. I agree with his statement that "making machine-readable > >> > tokens be spelled in different ways is a 'disease'." I'd like to > >> > leave this as-is so that the documentation can clearly state the exact > >> > set of allowable values. > >> > > >> > [1] https://lore.kernel.org/git/xmqqr1dqzyl7.fsf@gitster.g/ > >> > >> I think he's talking about batch, Batch, BATCH, bAtCh etc. there. But > >> the "maybe bool" is a stanard pattern we use. > >> > >> I don't think we'd call one of these 0, off, no or false etc. to avoid > >> confusion, so then you can use git_parse_maybe_...() > > > > I don't see the advantage of having multiple ways of specifying > > "none". The user can read the doc and know exactly what to write. If > > they write something unallowable, they get a clear warning and they > > can read the doc again to figure out what to write. This isn't a > > boolean options at all, so why should we entertain bool-like ways of > > spelling it? > > It's not boolean, it's multi-value and one of the values includes a true > or false boolean value. You just spell it "none". > > I think both this and your comment above suggest that you think there's > no point in this because you haven't interacted with/used "git config" > as a command line or API mechanism, but have just hand-crafted config > files. > > That's fair enough, but there's a lot of tooling that benefits from the > latter. My batch mode perf tests (on github, not yet submitted to the list) use `git -c core.fsync=<foo>` to set up a per-process config. I haven't used the `git config` writing support in a while, so I haven't deeply thought about that. However, I favor simplifying the use case of "atomically" setting a new holistic core.fsync value versus the use case of deriving a new core.fsync value from the preexisting value. > E.g.: > > $ git -c core.fsync=off config --type=bool core.fsync > false > $ git -c core.fsync=blah config --type=bool core.fsync > fatal: bad boolean config value 'blah' for 'core.fsync' > > Here we can get 'git config' to normalize what you call 'none', and you > can tell via exit codes/normalization if it's "false". But if you invent > a new term for "false" you can't do that as easily. > > We have various historical keys that take odd exceptions to that, > e.g. "never", but unless we have a good reason to let's not invent more > exceptions. > > >> >> > Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> > >> >> > --- > >> >> > Documentation/config/core.txt | 27 +++++++++---- > >> >> > builtin/fast-import.c | 2 +- > >> >> > builtin/index-pack.c | 4 +- > >> >> > builtin/pack-objects.c | 8 ++-- > >> >> > bulk-checkin.c | 5 ++- > >> >> > cache.h | 39 +++++++++++++++++- > >> >> > commit-graph.c | 3 +- > >> >> > config.c | 76 ++++++++++++++++++++++++++++++++++- > >> >> > csum-file.c | 5 ++- > >> >> > csum-file.h | 3 +- > >> >> > environment.c | 1 + > >> >> > midx.c | 3 +- > >> >> > object-file.c | 3 +- > >> >> > pack-bitmap-write.c | 3 +- > >> >> > pack-write.c | 13 +++--- > >> >> > read-cache.c | 2 +- > >> >> > 16 files changed, 164 insertions(+), 33 deletions(-) > >> >> > > >> >> > diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt > >> >> > index dbb134f7136..4f1747ec871 100644 > >> >> > --- a/Documentation/config/core.txt > >> >> > +++ b/Documentation/config/core.txt > >> >> > @@ -547,6 +547,25 @@ core.whitespace:: > >> >> > is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent` > >> >> > errors. The default tab width is 8. Allowed values are 1 to 63. > >> >> > > >> >> > +core.fsync:: > >> >> > + A comma-separated list of parts of the repository which should be > >> >> > + hardened via the core.fsyncMethod when created or modified. You can > >> >> > + disable hardening of any component by prefixing it with a '-'. Later > >> >> > + items take precedence over earlier ones in the list. For example, > >> >> > + `core.fsync=all,-pack-metadata` means "harden everything except pack > >> >> > + metadata." Items that are not hardened may be lost in the event of an > >> >> > + unclean system shutdown. > >> >> > ++ > >> >> > +* `none` disables fsync completely. This must be specified alone. > >> >> > +* `loose-object` hardens objects added to the repo in loose-object form. > >> >> > +* `pack` hardens objects added to the repo in packfile form. > >> >> > +* `pack-metadata` hardens packfile bitmaps and indexes. > >> >> > +* `commit-graph` hardens the commit graph file. > >> >> > +* `objects` is an aggregate option that includes `loose-objects`, `pack`, > >> >> > + `pack-metadata`, and `commit-graph`. > >> >> > +* `default` is an aggregate option that is equivalent to `objects,-loose-object` > >> >> > +* `all` is an aggregate option that syncs all individual components above. > >> >> > + > >> >> > >> >> It's probably a *bit* more work to set up, but I wonder if this wouldn't > >> >> be simpler if we just said (and this is partially going against what I > >> >> noted above): > >> >> > >> >> == BEGIN DOC > >> >> > >> >> core.fsync is a multi-value config variable where each item is a > >> >> pathspec that'll get matched the same way as 'git-ls-files' et al. > >> >> > >> >> When we sync pretend that a path like .git/objects/de/adbeef... is > >> >> relative to the top-level of the git > >> >> directory. E.g. "objects/de/adbeaf.." or "objects/pack/...". > >> >> > >> >> You can then supply a list of wildcards and exclusions to configure > >> >> syncing. or "false", "off" etc. to turn it off. These are synonymous > >> >> with: > >> >> > >> >> ; same as "false" > >> >> core.fsync = ":!*" > >> >> > >> >> Or: > >> >> > >> >> ; same as "true" > >> >> core.fsync = "*" > >> >> > >> >> Or, to selectively sync some things and not others: > >> >> > >> >> ;; Sync objects, but not "info" > >> >> core.fsync = ":!objects/info/**" > >> >> core.fsync = "objects/**" > >> >> > >> >> See gitrepository-layout(5) for details about what sort of paths you > >> >> might be expected to match. Not all paths listed there will go through > >> >> this mechanism (e.g. currently objects do, but nothing to do with config > >> >> does). > >> >> > >> >> We can and will match this against "fake paths", e.g. when writing out > >> >> packs we may match against just the string "objects/pack", we're not > >> >> going to re-check if every packfile we're writing matches your globs, > >> >> ditto for loose objects. Be reasonable! > >> >> > >> >> This metharism is intended as a shorthand that provides some flexibility > >> >> when fsyncing, while not forcing git to come up with labels for all > >> >> paths the git dir, or to support crazyness like "objects/de/adbeef*" > >> >> > >> >> More paths may be added or removed in the future, and we make no > >> >> promises that we won't move things around, so if in doubt use > >> >> e.g. "true" or a wide pattern match like "objects/**". When in doubt > >> >> stick to the golden path of examples provided in this documentation. > >> >> > >> >> == END DOC > >> >> > >> >> > >> >> It's a tad more complex to set up, but I wonder if that isn't worth > >> >> it. It nicely gets around any current and future issues of deciding what > >> >> labels such as "loose-object" etc. to pick, as well as slotting into an > >> >> existing method of doing exclude/include lists. > >> >> > >> > > >> > I think this proposal is a lot of complexity to avoid coming up with a > >> > new name for syncable things as they are added to Git. A path based > >> > mechanism makes it hard to document for the (advanced) user what the > >> > full set of things is and how it might change from release to release. > >> > I think the current core.fsync scheme is a bit easier to understand, > >> > query, and extend. > >> > >> We document it in gitrepository-layout(5). Yeah it has some > >> disadvantages, but one advantage is that you could make the > >> composability easy. I.e. if last exclude wins then a setting of: > >> > >> core.fsync = ":!*" > >> core.fsync = "objects/**" > >> > >> Would reset all previous matches & only match objects/**. > >> > > > > The value of changing this is predicated on taking your previous > > multi-valued config proposal, which I'm still not at all convinced > > about. > > They're orthagonal. I.e. you get benefits from multi-value with or > without this globbing mechanism. > > In any case, I don't feel strongly about/am really advocating this > globbing mechanism. I just wondered if it wouldn't make things simpler > since it would sidestep the need to create any sort of categories for > subsets of gitrepository-layout(5), but maybe not... > > > The schema in the current (v1-v2) version of the patch already > > includes an example of extending the list of syncable things, and > > Patrick Steinhardt made it clear that he feels comfortable adding > > 'refs' to the same schema in a future change. > > > > I'll also emphasize that we're talking about a non-functional, > > relatively corner-case behavioral configuration. These values don't > > change how git's interface behaves except when the system crashes > > during a git command or shortly after one completes. > > That may be something some OS's promise, but it's not something fsync() > or POSIX promises. I.e. you might write a ref, but unless you fsync and > the relevant dir entries another process might not see the update, crash > or not. > I haven't seen any indication that POSIX requires an fsync for visiblity within a running system. I looked at the doc for open, write, and fsync, and saw no indication that it's posix compliant to require an fsync for visibility. I think an OS that required fsync for cross-process visiblity would fail to run Git for a myriad of other reasons and would likely lose all its users. I'm curious where you've seen documentation that allows such unhelpful behavior? > That's an aside from these config design questions, and I think > most/(all?) OS's anyone cares about these days tend to make that > implicit promise as part of their VFS behavior, but we should probably > design such an interface to fsync() with such pedantic portability in > mind. Why? To be rude to such a hypothetical system, if a system were so insanely designed, it would be nuts to support it. > > While you may not personally love the proposed configuration > > interface, I'd want your view on some questions: > > 1. Is it easy for the (advanced) user to set a configuration? > > 2. Is it easy for the (advanced) user to see what was configured? > > 3. Is it easy for the Git community to build on this as we want to add > > things to the list of things to sync? > > a) Is there a good best practice configuration so that people can > > avoid losing integrity for new stuff that they are intending to sync. > > b) If someone has a custom configuration, can that custom > > configuration do something reasonable as they upgrade versions of Git? > > ** In response to this question, I might see some value > > in adding a 'derived-metadata' aggregate that can be disabled so that > > a custom configuration can exclude those as they change version to > > version. > > c) Is it too much maintenance overhead to consider how to present > > this configuration knob for any new hashfile or other datafile in the > > git repo? > > 4. Is there a good path forward to change the default syncable set, > > both in git-for-windows and in Git for other platforms? > > I'm not really sure this globbing this is a good idea, as noted above > just a suggestion etc. > > As noted there it just gets you out of the business of re-defining > gitrepository-layout(5), and assuming too much in advance about certain > use-cases. > > E.g. even "refs" might be too broad for some. I don't tend to be I/O > limited, but I could see how someone who would be would care about > refs/heads but not refs/remotes, or want to exclude logs/* but not the > refs updates themselves etc. This use-case is interesting (distinguishing remote refs from local refs). I think the difficulty of verifying (for even an advanced user) that the right fsyncing is actually happening still puts me on the side of having a carefully curated and documented set of syncable things rather than a file-path-based mechanism. Is this meaningful in the presumably nearby future world of the refsdb backend? Is that somehow split by remote versus local? > >> >> > diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c > >> >> > index 857be7826f3..916c55d6ce9 100644 > >> >> > --- a/builtin/pack-objects.c > >> >> > +++ b/builtin/pack-objects.c > >> >> > @@ -1204,11 +1204,13 @@ static void write_pack_file(void) > >> >> > * If so, rewrite it like in fast-import > >> >> > */ > >> >> > if (pack_to_stdout) { > >> >> > - finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_CLOSE); > >> >> > + finalize_hashfile(f, hash, FSYNC_COMPONENT_NONE, > >> >> > + CSUM_HASH_IN_STREAM | CSUM_CLOSE); > >> >> > >> >> Not really related to this per-se, but since you're touching the API > >> >> everything goes through I wonder if callers should just always try to > >> >> fsync, and we can just catch EROFS and EINVAL in the wrapper if someone > >> >> tries to flush stdout, or catch the fd at that lower level. > >> >> > >> >> Or maybe there's a good reason for this... > >> > > >> > It's platform dependent, but I'd expect fsync would do something for > >> > pipes or stdout redirected to a file. In these cases we really don't > >> > want to fsync since we have no idea what we're talking to and we're > >> > potentially worsening performance for probably no benefit. > >> > >> Yeah maybe we should just leave it be. > >> > >> I'd think the C library returning EINVAL would be a trivial performance > >> cost though. > >> > >> It just seemed odd to hardcode assumptions about what can and can't be > >> synced when the POSIX defined function will also tell us that. > >> > > > > Redirecting stdout to a file seems like a common usage for this > > command. That would definitely be fsyncable, but Git has no idea what > > its proper category is since there's no way to know the purpose or > > lifetime of the packfile. I'm going to leave this be, because I'd > > posit that "can it be fsynced?" is not the same as "should it be > > fsynced?". The latter question can't be answered for stdout. > > As noted this was just an aside, and I don't even know if any OS would > do anything meaningful with an fsync() of such a FD anyway. > The underlying fsync primitive does have a meaning on Windows for pipes, but it's certainly not what Git would want to do. Also if stdout is redirected to a file, I'm pretty sure that UNIX OSes would respect the fsync call. However it's not meaningful in the sense of the git repository, since we don't know what the packfile is or why it was created. > I just don't see why we wouldn't say: > > 1. We're syncing this category of thing > 2. Try it > 3. If fsync returns "can't fsync that sort of thing" move on > > As opposed to trying to shortcut #3 by doing the detection ourselves. > > I.e. maybe there was a good reason, but it seemed to be some easy > potential win for more simplification, since you were re-doing and > simplifying some of the interface anyway... We're trying to be deliberate about what we're fsyncing. Fsyncing an unknown file created by the packfile code doesn't move us in that direction. In your taxonomy we don't know (1), "what is this category of thing?" Sure it's got the packfile format, but is not known to be an actual packfile that's part of the repository. > >> > >> >> > [...] > >> >> > +/* > >> >> > + * These values are used to help identify parts of a repository to fsync. > >> >> > + * FSYNC_COMPONENT_NONE identifies data that will not be a persistent part of the > >> >> > + * repository and so shouldn't be fsynced. > >> >> > + */ > >> >> > +enum fsync_component { > >> >> > + FSYNC_COMPONENT_NONE = 0, > >> >> > >> >> I haven't read ahead much but in most other such cases we don't define > >> >> the "= 0", just start at 1<<0, then check the flags elsewhere... > >> >> > >> >> > +static const struct fsync_component_entry { > >> >> > + const char *name; > >> >> > + enum fsync_component component_bits; > >> >> > +} fsync_component_table[] = { > >> >> > + { "loose-object", FSYNC_COMPONENT_LOOSE_OBJECT }, > >> >> > + { "pack", FSYNC_COMPONENT_PACK }, > >> >> > + { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA }, > >> >> > + { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, > >> >> > + { "objects", FSYNC_COMPONENTS_OBJECTS }, > >> >> > + { "default", FSYNC_COMPONENTS_DEFAULT }, > >> >> > + { "all", FSYNC_COMPONENTS_ALL }, > >> >> > +}; > >> >> > + > >> >> > +static enum fsync_component parse_fsync_components(const char *var, const char *string) > >> >> > +{ > >> >> > + enum fsync_component output = 0; > >> >> > + > >> >> > + if (!strcmp(string, "none")) > >> >> > + return output; > >> >> > + > >> >> > + while (string) { > >> >> > + int i; > >> >> > + size_t len; > >> >> > + const char *ep; > >> >> > + int negated = 0; > >> >> > + int found = 0; > >> >> > + > >> >> > + string = string + strspn(string, ", \t\n\r"); > >> >> > >> >> Aside from the "use a list" isn't this hardcoding some windows-specific > >> >> assumptions with \n\r? Maybe not... > >> > > >> > I shamelessly stole this code from parse_whitespace_rule. I thought > >> > about making a helper to be called by both functions, but the amount > >> > of state going into and out of the wrapper via arguments was > >> > substantial and seemed to negate the benefit of deduplication. > >> > >> FWIW string_list_split() is easier to work with in those cases, or at > >> least I think so... > > > > This code runs at startup for a variable that may be present on some > > installations. The nice property of the current patch's code is that > > it's already a well-tested pattern that doesn't do any allocations as > > it's working, unlike string_list_split(). > > Multi-value config would also get you fewer allocations :) > > Anyway, I mainly meant to point out that for stuff like this it's fine > to optimize it for ease rather than micro-optimize allocations. Those > really aren't a bottleneck on this scale. > > Even in that case there's string_list_split_in_place(), which can be a > bit nicer than manual C-string fiddling. > Am I allowed to change the config value string in place? The core.whitespace code is careful not to modify the string. I kind of like the parse_ws_error_highlight code a little better now that I've seen it, but I think the current code is fine too. > > I hope you know that I appreciate your review feedback, even though > > I'm pushing back on most of it so far this round. I'll be sending v3 > > to the list soon after giving it another look over. > > Sure, no worries. Just hoping to help. If you go for something different > etc. that's fine. Just hoping to bridge the gap in some knowledge / > offer potentially interesting suggestions (some of which may be dead > ends, like the config glob thing...). Thanks, Neeraj ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v2 2/3] core.fsync: introduce granular fsync control 2021-12-09 6:18 ` Neeraj Singh @ 2022-01-18 23:50 ` Neeraj Singh 2022-01-19 15:28 ` Ævar Arnfjörð Bjarmason 2022-01-19 14:52 ` Ævar Arnfjörð Bjarmason 1 sibling, 1 reply; 122+ messages in thread From: Neeraj Singh @ 2022-01-18 23:50 UTC (permalink / raw) To: Ævar Arnfjörð Bjarmason Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Patrick Steinhardt, Neeraj Singh Hi Ævar, Could you please respond to the parent message? To summarize the main points and questions: 1) The current patch series design of core.fsync favors ease of modification of the complete configuration as a single atomic config step, at the cost of making it somewhat harder to derive a new configuration from an existing one. See [1] where this facility is used. 2) Is there any existing configuration that uses the multi-value schema you proposed? The diff.wsErrorHighlight setting is actually comma-separated. 3) Is there any system you can point at or any support in the POSIX spec for requiring fsync for anything other than durability of data across system crash? 4) I think string_list_split_in_place is not valid for splitting a comma-separated list in the config code, since the value coming from the configset code is in some global data structure that might be used again. It seems like we could have subtle problems down the line if we mutate it. [1] https://github.com/neerajsi-msft/git/commit/7e9749a7e94d26c88789459588997329c5130792#diff-ee0307657f5a76b723c8973db0dbd5a2ca62e14b02711b897418b35d78fc6023R1327 Thanks, Neeraj ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v2 2/3] core.fsync: introduce granular fsync control 2022-01-18 23:50 ` Neeraj Singh @ 2022-01-19 15:28 ` Ævar Arnfjörð Bjarmason 0 siblings, 0 replies; 122+ messages in thread From: Ævar Arnfjörð Bjarmason @ 2022-01-19 15:28 UTC (permalink / raw) To: Neeraj Singh Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Patrick Steinhardt, Neeraj Singh On Tue, Jan 18 2022, Neeraj Singh wrote: > Hi Ævar, > Could you please respond to the parent message? I did just now in https://lore.kernel.org/git/220119.8635ljoidt.gmgdl@evledraar.gmail.com/; sorry about the delay. This thread fell off my radar. > To summarize the main points and questions: > 1) The current patch series design of core.fsync favors ease of > modification of the complete configuration as a single atomic config > step, at the cost of making it somewhat harder to derive a new > configuration from an existing one. See [1] where this facility is > used. > > 2) Is there any existing configuration that uses the multi-value > schema you proposed? The diff.wsErrorHighlight setting is actually > comma-separated. I replied to that. To add a bit to that the comma-delimited thing isn't any sort of a "blocker" or whatever in my mind. I just wanted to point out that you could get the same with multi-value with some better integration, and we might have our cake & eat it too. But at the end of the day if you disagree you're doing the work, and that's ultimately bikeshedding of the config interface. I'm fine with it either way. > 3) Is there any system you can point at or any support in the POSIX > spec for requiring fsync for anything other than durability of data > across system crash? Some examples in the linked... > 4) I think string_list_split_in_place is not valid for splitting a > comma-separated list in the config code, since the value coming from > the configset code is in some global data structure that might be used > again. It seems like we could have subtle problems down the line if we > mutate it. Replied to that too, hope it's all useful. Thanks again for working on this, it's really nice to have someone move the state of data integrity in git forward. > [1] https://github.com/neerajsi-msft/git/commit/7e9749a7e94d26c88789459588997329c5130792#diff-ee0307657f5a76b723c8973db0dbd5a2ca62e14b02711b897418b35d78fc6023R1327 ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v2 2/3] core.fsync: introduce granular fsync control 2021-12-09 6:18 ` Neeraj Singh 2022-01-18 23:50 ` Neeraj Singh @ 2022-01-19 14:52 ` Ævar Arnfjörð Bjarmason 2022-01-28 1:28 ` Neeraj Singh 1 sibling, 1 reply; 122+ messages in thread From: Ævar Arnfjörð Bjarmason @ 2022-01-19 14:52 UTC (permalink / raw) To: Neeraj Singh Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Patrick Steinhardt, Neeraj Singh On Wed, Dec 08 2021, Neeraj Singh wrote: [Sorry about the late reply, and thanks for the downthread prodding] > On Wed, Dec 8, 2021 at 8:46 PM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote: >> >> >> On Wed, Dec 08 2021, Neeraj Singh wrote: >> >> > On Wed, Dec 8, 2021 at 2:17 AM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote: >> >> >> >> >> >> On Tue, Dec 07 2021, Neeraj Singh wrote: >> >> >> >> > On Tue, Dec 7, 2021 at 5:01 AM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote: >> >> >> >> >> >> >> >> >> On Tue, Dec 07 2021, Neeraj Singh via GitGitGadget wrote: >> >> >> >> >> >> > From: Neeraj Singh <neerajsi@microsoft.com> >> >> >> > >> >> >> > This commit introduces the `core.fsync` configuration >> >> >> > knob which can be used to control how components of the >> >> >> > repository are made durable on disk. >> >> >> > >> >> >> > This setting allows future extensibility of the list of >> >> >> > syncable components: >> >> >> > * We issue a warning rather than an error for unrecognized >> >> >> > components, so new configs can be used with old Git versions. >> >> >> >> >> >> Looks good! >> >> >> >> >> >> > * We support negation, so users can choose one of the default >> >> >> > aggregate options and then remove components that they don't >> >> >> > want. The user would then harden any new components added in >> >> >> > a Git version update. >> >> >> >> >> >> I think this config schema makes sense, but just a (I think important) >> >> >> comment on the "how" not "what" of it. It's really much better to define >> >> >> config as: >> >> >> >> >> >> [some] >> >> >> key = value >> >> >> key = value2 >> >> >> >> >> >> Than: >> >> >> >> >> >> [some] >> >> >> key = value,value2 >> >> >> >> >> >> The reason is that "git config" has good support for working with >> >> >> multi-valued stuff, so you can do e.g.: >> >> >> >> >> >> git config --get-all -z some.key >> >> >> >> >> >> And you can easily (re)set such config e.g. with --replace-all etc., but >> >> >> for comma-delimited you (and users) need to do all that work themselves. >> >> >> >> >> >> Similarly instead of: >> >> >> >> >> >> some.key = want-this >> >> >> some.key = -not-this >> >> >> some.key = but-want-this >> >> >> >> >> >> I think it's better to just have two lists, one inclusive another >> >> >> exclusive. E.g. see "log.decorate" and "log.excludeDecoration", >> >> >> "transfer.hideRefs" >> >> >> >> >> >> Which would mean: >> >> >> >> >> >> core.fsync = want-this >> >> >> core.fsyncExcludes = -not-this >> >> >> >> >> >> For some value of "fsyncExcludes", maybe "noFsync"? Anyway, just a >> >> >> suggestion on making this easier for users & the implementation. >> >> >> >> >> > >> >> > Maybe there's some way to handle this I'm unaware of, but a >> >> > disadvantage of your multi-valued config proposal is that it's harder, >> >> > for example, for a per-repo config store to reasonably override a >> >> > per-user config store. With the configuration scheme as-is, I can >> >> > have a per-user setting like `core.fsync=all` which covers my typical >> >> > repos, but then have a maintainer repo with a private setting of >> >> > `core.fsync=none` to speed up cases where I'm mostly working with >> >> > other people's changes that are backed up in email or server-side >> >> > repos. The latter setting conveniently overrides the former setting >> >> > in all aspects. >> >> >> >> Even if you turn just your comma-delimited proposal into a list proposal >> >> can't we just say that the last one wins? Then it won't matter what cmae >> >> before, you'd specify "core.fsync=none" in your local .git/config. >> >> >> >> But this is also a general issue with a bunch of things in git's config >> >> space. I'd rather see us use the list-based values and just come up with >> >> some general way to reset them that works for all keys, rather than >> >> regretting having comma-delimited values that'll be harder to work with >> >> & parse, which will be a legacy wart if/when we come up with a way to >> >> say "reset all previous settings". >> >> >> >> > Also, with the core.fsync and core.fsyncExcludes, how would you spell >> >> > "don't sync anything"? Would you still have the aggregate options.? >> >> >> >> core.fsyncExcludes = * >> >> >> >> I.e. the same as the core.fsync=none above, anyway I still like the >> >> wildcard thing below a bit more... >> > >> > I'm not going to take this feedback unless there are additional votes >> > from the Git community in this direction. I make the claim that >> > single-valued comma-separated config lists are easier to work with in >> > the existing Git infrastructure. >> >> Easier in what sense? I showed examples of how "git-config" trivially >> works with multi-valued config, but for comma-delimited you'll need to >> write your own shellscript boilerplate around simple things like adding >> values, removing existing ones etc. >> >> I.e. you can use --add, --unset, --unset-all, --get, --get-all etc. >> > > I see what you're saying for cases where someone would want to set a > core.fsync setting that's derived from the user's current config. But > I'm guessing that the dominant use case is someone setting a new fsync > configuration that achieves some atomic goal with respect to a given > repository. Like "this is a throwaway, so sync nothing" or "this is > really important, so sync all objects and refs and the index". Whether it's multi-value or comma-separated you could do: -c core.fsync=[none|false] To sync nothing, i.e. if we see "none/false" it doesn't matter if we saw core.fsync=loose-object or whatever before, it would override it, ditto for "all". >> > parsing code for the core.whitespace variable and users are used to >> > this syntax there. There are several other comma-separated lists in >> > the config space, so this construct has precedence and will be with >> > Git for some time. >> >> That's not really an argument either way for why we'd pick X over Y for >> something new. We've got some comma-delimited, some multi-value (I'm >> fairly sure we have more multi-value). >> > > My main point here is that there's precedent for patch's current exact > schema for a config in the same core config leaf of the Documentation. > It seems from your comments that we'd have to invent and document some > new convention for "reset" of a multi-valued config. So you're > suggesting that I solve an extra set of problems to get this change > in. Just want to remind you that my personal itch to scratch is to > get the underlying mechanism in so that git-for-windows can set its > default setting to batch mode. I'm not expecting many users to > actually configure this setting to any non-default value. Me neither. I think people will most likely set this once in ~/.gitconfig or /etc/gitconfig. We have some config keys that are multi-value and either comma-separated or space-separated, e.g. core.alternateRefsPrefixes Then we have e.g. blame.ignoreRevsFile which is multi-value, and further has the convention that setting it to an empty value clears the list. which would scratch the "override existing" itch. format.notes, versionsort.suffix, transfer.hideRefs, branch.<name>.merge are exmples of existing multi-value config. >> > Also, fsync configurations aren't composable like >> > some other configurations may be. It makes sense to have a holistic >> > singular fsync configuration, which is best represented by a single >> > variable. >> >> What's a "variable" here? We call these "keys", you can have a >> single-value key like user.name that you get with --get, or a >> multi-value key like say branch.<name>.merge or push.pushOption that >> you'd get with --get-all. > > Yeah, I meant "key". I conflated the config key with the underlying > global variable in git. *nod* >> I think you may be referring to either not wanting these to be >> "inherited" (which is not really a think we do for anything else in >> config), or lacking the ability to "reset". >> >> For the latter if that's absolutely needed we could just use the same >> trick as "diff.wsErrorHighlight" uses of making an empty value "reset" >> the list, and you'd get better "git config" support for editing it. >> > > My reading of the code is that diff.wsErrorHighlight is a comma > separated list and not a multi-valued config. Actually I haven't yet > found an existing multi-valued config (not sure how to grep for it). Yes, I think I conflated it with one of the ones above when I wrote this. >> >> >> > This also supports the common request of doing absolutely no >> >> >> > fysncing with the `core.fsync=none` value, which is expected >> >> >> > to make the test suite faster. >> >> >> >> >> >> Let's just use the git_parse_maybe_bool() or git_parse_maybe_bool_text() >> >> >> so we'll accept "false", "off", "no" like most other such config? >> >> > >> >> > Junio's previous feedback when discussing batch mode [1] was to offer >> >> > less flexibility when parsing new values of these configuration >> >> > options. I agree with his statement that "making machine-readable >> >> > tokens be spelled in different ways is a 'disease'." I'd like to >> >> > leave this as-is so that the documentation can clearly state the exact >> >> > set of allowable values. >> >> > >> >> > [1] https://lore.kernel.org/git/xmqqr1dqzyl7.fsf@gitster.g/ >> >> >> >> I think he's talking about batch, Batch, BATCH, bAtCh etc. there. But >> >> the "maybe bool" is a stanard pattern we use. >> >> >> >> I don't think we'd call one of these 0, off, no or false etc. to avoid >> >> confusion, so then you can use git_parse_maybe_...() >> > >> > I don't see the advantage of having multiple ways of specifying >> > "none". The user can read the doc and know exactly what to write. If >> > they write something unallowable, they get a clear warning and they >> > can read the doc again to figure out what to write. This isn't a >> > boolean options at all, so why should we entertain bool-like ways of >> > spelling it? >> >> It's not boolean, it's multi-value and one of the values includes a true >> or false boolean value. You just spell it "none". >> >> I think both this and your comment above suggest that you think there's >> no point in this because you haven't interacted with/used "git config" >> as a command line or API mechanism, but have just hand-crafted config >> files. >> >> That's fair enough, but there's a lot of tooling that benefits from the >> latter. > > My batch mode perf tests (on github, not yet submitted to the list) > use `git -c core.fsync=<foo>` to set up a per-process config. I > haven't used the `git config` writing support in a while, so I haven't > deeply thought about that. However, I favor simplifying the use case > of "atomically" setting a new holistic core.fsync value versus the use > case of deriving a new core.fsync value from the preexisting value. If you implement it like blame.ignoreRevsFile you can have your cake and eat it too, i.e.: -c core.fsync= core.fsync=loose-object ensures only loose objects are synced, as with your single-value config, but I'd think what you'd be more likely to actually mean would be: # With "core.fsync=pack" set in ~/.gitconfig -c core.fsync=loose-object I.e. that the common case is "I want this to be synced here", not that you'd like to declare sync policy from scratch every time. In any case, on this general topic my main point is that the git-config(1) command has pretty integration for multi-value if you do it that way, and not for comma-delimited. I.e. you get --add, --unset, --unset-all, --get, --get-all etc. So I think for anything new it makes sense to lean into that, I think most of these existing comma-delimited ones are ones we'd do differently today on reflection. And if you suppor the "empty resets" like blame.ignoreRevsFile it seems to me you'll have your cake & eat it too. >> E.g.: >> >> $ git -c core.fsync=off config --type=bool core.fsync >> false >> $ git -c core.fsync=blah config --type=bool core.fsync >> fatal: bad boolean config value 'blah' for 'core.fsync' >> >> Here we can get 'git config' to normalize what you call 'none', and you >> can tell via exit codes/normalization if it's "false". But if you invent >> a new term for "false" you can't do that as easily. >> >> We have various historical keys that take odd exceptions to that, >> e.g. "never", but unless we have a good reason to let's not invent more >> exceptions. >> >> >> >> > Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> >> >> >> > --- >> >> >> > Documentation/config/core.txt | 27 +++++++++---- >> >> >> > builtin/fast-import.c | 2 +- >> >> >> > builtin/index-pack.c | 4 +- >> >> >> > builtin/pack-objects.c | 8 ++-- >> >> >> > bulk-checkin.c | 5 ++- >> >> >> > cache.h | 39 +++++++++++++++++- >> >> >> > commit-graph.c | 3 +- >> >> >> > config.c | 76 ++++++++++++++++++++++++++++++++++- >> >> >> > csum-file.c | 5 ++- >> >> >> > csum-file.h | 3 +- >> >> >> > environment.c | 1 + >> >> >> > midx.c | 3 +- >> >> >> > object-file.c | 3 +- >> >> >> > pack-bitmap-write.c | 3 +- >> >> >> > pack-write.c | 13 +++--- >> >> >> > read-cache.c | 2 +- >> >> >> > 16 files changed, 164 insertions(+), 33 deletions(-) >> >> >> > >> >> >> > diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt >> >> >> > index dbb134f7136..4f1747ec871 100644 >> >> >> > --- a/Documentation/config/core.txt >> >> >> > +++ b/Documentation/config/core.txt >> >> >> > @@ -547,6 +547,25 @@ core.whitespace:: >> >> >> > is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent` >> >> >> > errors. The default tab width is 8. Allowed values are 1 to 63. >> >> >> > >> >> >> > +core.fsync:: >> >> >> > + A comma-separated list of parts of the repository which should be >> >> >> > + hardened via the core.fsyncMethod when created or modified. You can >> >> >> > + disable hardening of any component by prefixing it with a '-'. Later >> >> >> > + items take precedence over earlier ones in the list. For example, >> >> >> > + `core.fsync=all,-pack-metadata` means "harden everything except pack >> >> >> > + metadata." Items that are not hardened may be lost in the event of an >> >> >> > + unclean system shutdown. >> >> >> > ++ >> >> >> > +* `none` disables fsync completely. This must be specified alone. >> >> >> > +* `loose-object` hardens objects added to the repo in loose-object form. >> >> >> > +* `pack` hardens objects added to the repo in packfile form. >> >> >> > +* `pack-metadata` hardens packfile bitmaps and indexes. >> >> >> > +* `commit-graph` hardens the commit graph file. >> >> >> > +* `objects` is an aggregate option that includes `loose-objects`, `pack`, >> >> >> > + `pack-metadata`, and `commit-graph`. >> >> >> > +* `default` is an aggregate option that is equivalent to `objects,-loose-object` >> >> >> > +* `all` is an aggregate option that syncs all individual components above. >> >> >> > + >> >> >> >> >> >> It's probably a *bit* more work to set up, but I wonder if this wouldn't >> >> >> be simpler if we just said (and this is partially going against what I >> >> >> noted above): >> >> >> >> >> >> == BEGIN DOC >> >> >> >> >> >> core.fsync is a multi-value config variable where each item is a >> >> >> pathspec that'll get matched the same way as 'git-ls-files' et al. >> >> >> >> >> >> When we sync pretend that a path like .git/objects/de/adbeef... is >> >> >> relative to the top-level of the git >> >> >> directory. E.g. "objects/de/adbeaf.." or "objects/pack/...". >> >> >> >> >> >> You can then supply a list of wildcards and exclusions to configure >> >> >> syncing. or "false", "off" etc. to turn it off. These are synonymous >> >> >> with: >> >> >> >> >> >> ; same as "false" >> >> >> core.fsync = ":!*" >> >> >> >> >> >> Or: >> >> >> >> >> >> ; same as "true" >> >> >> core.fsync = "*" >> >> >> >> >> >> Or, to selectively sync some things and not others: >> >> >> >> >> >> ;; Sync objects, but not "info" >> >> >> core.fsync = ":!objects/info/**" >> >> >> core.fsync = "objects/**" >> >> >> >> >> >> See gitrepository-layout(5) for details about what sort of paths you >> >> >> might be expected to match. Not all paths listed there will go through >> >> >> this mechanism (e.g. currently objects do, but nothing to do with config >> >> >> does). >> >> >> >> >> >> We can and will match this against "fake paths", e.g. when writing out >> >> >> packs we may match against just the string "objects/pack", we're not >> >> >> going to re-check if every packfile we're writing matches your globs, >> >> >> ditto for loose objects. Be reasonable! >> >> >> >> >> >> This metharism is intended as a shorthand that provides some flexibility >> >> >> when fsyncing, while not forcing git to come up with labels for all >> >> >> paths the git dir, or to support crazyness like "objects/de/adbeef*" >> >> >> >> >> >> More paths may be added or removed in the future, and we make no >> >> >> promises that we won't move things around, so if in doubt use >> >> >> e.g. "true" or a wide pattern match like "objects/**". When in doubt >> >> >> stick to the golden path of examples provided in this documentation. >> >> >> >> >> >> == END DOC >> >> >> >> >> >> >> >> >> It's a tad more complex to set up, but I wonder if that isn't worth >> >> >> it. It nicely gets around any current and future issues of deciding what >> >> >> labels such as "loose-object" etc. to pick, as well as slotting into an >> >> >> existing method of doing exclude/include lists. >> >> >> >> >> > >> >> > I think this proposal is a lot of complexity to avoid coming up with a >> >> > new name for syncable things as they are added to Git. A path based >> >> > mechanism makes it hard to document for the (advanced) user what the >> >> > full set of things is and how it might change from release to release. >> >> > I think the current core.fsync scheme is a bit easier to understand, >> >> > query, and extend. >> >> >> >> We document it in gitrepository-layout(5). Yeah it has some >> >> disadvantages, but one advantage is that you could make the >> >> composability easy. I.e. if last exclude wins then a setting of: >> >> >> >> core.fsync = ":!*" >> >> core.fsync = "objects/**" >> >> >> >> Would reset all previous matches & only match objects/**. >> >> >> > >> > The value of changing this is predicated on taking your previous >> > multi-valued config proposal, which I'm still not at all convinced >> > about. >> >> They're orthagonal. I.e. you get benefits from multi-value with or >> without this globbing mechanism. >> >> In any case, I don't feel strongly about/am really advocating this >> globbing mechanism. I just wondered if it wouldn't make things simpler >> since it would sidestep the need to create any sort of categories for >> subsets of gitrepository-layout(5), but maybe not... >> >> > The schema in the current (v1-v2) version of the patch already >> > includes an example of extending the list of syncable things, and >> > Patrick Steinhardt made it clear that he feels comfortable adding >> > 'refs' to the same schema in a future change. >> > >> > I'll also emphasize that we're talking about a non-functional, >> > relatively corner-case behavioral configuration. These values don't >> > change how git's interface behaves except when the system crashes >> > during a git command or shortly after one completes. >> >> That may be something some OS's promise, but it's not something fsync() >> or POSIX promises. I.e. you might write a ref, but unless you fsync and >> the relevant dir entries another process might not see the update, crash >> or not. >> > > I haven't seen any indication that POSIX requires an fsync for > visiblity within a running system. I looked at the doc for open, > write, and fsync, and saw no indication that it's posix compliant to > require an fsync for visibility. I think an OS that required fsync > for cross-process visiblity would fail to run Git for a myriad of > other reasons and would likely lose all its users. I'm curious where > you've seen documentation that allows such unhelpful behavior? There's multiple unrelated and related things in this area. One is a case where you'll e.g. write a file "foo" using stdio, spawn a program to work on it in the same program, but it might not see it at all, or see empty content, the latter being because you haven't flushed your I/O buffers (which you can do via fsync()). The former is that on *nix systems you're generally only guaranteed to write to a fd, but not to have the associated metadata be synced for you. That is spelled out e.g. in the DESCRIPTION section of linux's fsync() manpage: https://man7.org/linux/man-pages/man2/fdatasync.2.html I don't know how much you follow non-Windows FS development, but there was also a very well known "incident" early in ext4 where it leaned into some permissive-by-POSIX behavior that caused data loss in practice on buggy programs that didn't correctly use fsync() , since various tooling had come to expect the stricter behavior of ext3: https://lwn.net/Articles/328363/ That was explicitly in the area of fs metadata being discussed here. Generally you can expect your VFS layer to be forgiving when it comes to IPC, but even that is out the window when it comes to networked filesystems, e.g. a shared git repository hosted on NFS. >> That's an aside from these config design questions, and I think >> most/(all?) OS's anyone cares about these days tend to make that >> implicit promise as part of their VFS behavior, but we should probably >> design such an interface to fsync() with such pedantic portability in >> mind. > > Why? To be rude to such a hypothetical system, if a system were so > insanely designed, it would be nuts to support it. Because we know that right now the system calls we're invoking aren't guaranteed to store data persistently to disk portably, although they do so in practice on most modern OS's. We're portably to a lot of platforms, and also need to keep e.g. NFS in mind, so being able to ask for a pedantic mode when you care about data retention at the cost of performance would be nice. And because the fsync config mode you're proposing is thoroughly non-standard, but is known to me much faster by leaning into known attributes of specific FS's on specific OS's, if we're not running on those it would be sensible to fall back to a stricter mode of operation. E.g. syncing all 100 loose objects we just wrote, not just the last one. >> > While you may not personally love the proposed configuration >> > interface, I'd want your view on some questions: >> > 1. Is it easy for the (advanced) user to set a configuration? >> > 2. Is it easy for the (advanced) user to see what was configured? >> > 3. Is it easy for the Git community to build on this as we want to add >> > things to the list of things to sync? >> > a) Is there a good best practice configuration so that people can >> > avoid losing integrity for new stuff that they are intending to sync. >> > b) If someone has a custom configuration, can that custom >> > configuration do something reasonable as they upgrade versions of Git? >> > ** In response to this question, I might see some value >> > in adding a 'derived-metadata' aggregate that can be disabled so that >> > a custom configuration can exclude those as they change version to >> > version. >> > c) Is it too much maintenance overhead to consider how to present >> > this configuration knob for any new hashfile or other datafile in the >> > git repo? >> > 4. Is there a good path forward to change the default syncable set, >> > both in git-for-windows and in Git for other platforms? >> >> I'm not really sure this globbing this is a good idea, as noted above >> just a suggestion etc. >> >> As noted there it just gets you out of the business of re-defining >> gitrepository-layout(5), and assuming too much in advance about certain >> use-cases. >> >> E.g. even "refs" might be too broad for some. I don't tend to be I/O >> limited, but I could see how someone who would be would care about >> refs/heads but not refs/remotes, or want to exclude logs/* but not the >> refs updates themselves etc. > > This use-case is interesting (distinguishing remote refs from local > refs). I think the difficulty of verifying (for even an advanced > user) that the right fsyncing is actually happening still puts me on > the side of having a carefully curated and documented set of syncable > things rather than a file-path-based mechanism. > > Is this meaningful in the presumably nearby future world of the refsdb > backend? Is that somehow split by remote versus local? There is the upcoming "reftable" work, but that's probably 2-3 years out at the earliest for series production workloads in git.git. >> >> >> > diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c >> >> >> > index 857be7826f3..916c55d6ce9 100644 >> >> >> > --- a/builtin/pack-objects.c >> >> >> > +++ b/builtin/pack-objects.c >> >> >> > @@ -1204,11 +1204,13 @@ static void write_pack_file(void) >> >> >> > * If so, rewrite it like in fast-import >> >> >> > */ >> >> >> > if (pack_to_stdout) { >> >> >> > - finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_CLOSE); >> >> >> > + finalize_hashfile(f, hash, FSYNC_COMPONENT_NONE, >> >> >> > + CSUM_HASH_IN_STREAM | CSUM_CLOSE); >> >> >> >> >> >> Not really related to this per-se, but since you're touching the API >> >> >> everything goes through I wonder if callers should just always try to >> >> >> fsync, and we can just catch EROFS and EINVAL in the wrapper if someone >> >> >> tries to flush stdout, or catch the fd at that lower level. >> >> >> >> >> >> Or maybe there's a good reason for this... >> >> > >> >> > It's platform dependent, but I'd expect fsync would do something for >> >> > pipes or stdout redirected to a file. In these cases we really don't >> >> > want to fsync since we have no idea what we're talking to and we're >> >> > potentially worsening performance for probably no benefit. >> >> >> >> Yeah maybe we should just leave it be. >> >> >> >> I'd think the C library returning EINVAL would be a trivial performance >> >> cost though. >> >> >> >> It just seemed odd to hardcode assumptions about what can and can't be >> >> synced when the POSIX defined function will also tell us that. >> >> >> > >> > Redirecting stdout to a file seems like a common usage for this >> > command. That would definitely be fsyncable, but Git has no idea what >> > its proper category is since there's no way to know the purpose or >> > lifetime of the packfile. I'm going to leave this be, because I'd >> > posit that "can it be fsynced?" is not the same as "should it be >> > fsynced?". The latter question can't be answered for stdout. >> >> As noted this was just an aside, and I don't even know if any OS would >> do anything meaningful with an fsync() of such a FD anyway. >> > > The underlying fsync primitive does have a meaning on Windows for > pipes, but it's certainly not what Git would want to do. Also if > stdout is redirected to a file, I'm pretty sure that UNIX OSes would > respect the fsync call. However it's not meaningful in the sense of > the git repository, since we don't know what the packfile is or why it > was created. I suggested that because I think it's probably nonsensical, but it's nonsense that POSIX seems to explicitly tell us that it'll handle (probably by silently doing nothing). So in terms of our interface we could lean into that and avoid our own special-casing. >> I just don't see why we wouldn't say: >> >> 1. We're syncing this category of thing >> 2. Try it >> 3. If fsync returns "can't fsync that sort of thing" move on >> >> As opposed to trying to shortcut #3 by doing the detection ourselves. >> >> I.e. maybe there was a good reason, but it seemed to be some easy >> potential win for more simplification, since you were re-doing and >> simplifying some of the interface anyway... > > We're trying to be deliberate about what we're fsyncing. Fsyncing an > unknown file created by the packfile code doesn't move us in that > direction. In your taxonomy we don't know (1), "what is this category > of thing?" Sure it's got the packfile format, but is not known to be > an actual packfile that's part of the repository. We know it's a fd, isn't that sufficient? In any case, I'm fine with also keeping it as is, I don't mean to split hairs here. It just stuck out as an odd part of the interface, why treat some fd's specially, instead of just throwing it all at the OS. Presumably the first thing the OS will do is figure out if it's a syncable fd or not, and act appropriately. >> >> >> >> >> > [...] >> >> >> > +/* >> >> >> > + * These values are used to help identify parts of a repository to fsync. >> >> >> > + * FSYNC_COMPONENT_NONE identifies data that will not be a persistent part of the >> >> >> > + * repository and so shouldn't be fsynced. >> >> >> > + */ >> >> >> > +enum fsync_component { >> >> >> > + FSYNC_COMPONENT_NONE = 0, >> >> >> >> >> >> I haven't read ahead much but in most other such cases we don't define >> >> >> the "= 0", just start at 1<<0, then check the flags elsewhere... >> >> >> >> >> >> > +static const struct fsync_component_entry { >> >> >> > + const char *name; >> >> >> > + enum fsync_component component_bits; >> >> >> > +} fsync_component_table[] = { >> >> >> > + { "loose-object", FSYNC_COMPONENT_LOOSE_OBJECT }, >> >> >> > + { "pack", FSYNC_COMPONENT_PACK }, >> >> >> > + { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA }, >> >> >> > + { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, >> >> >> > + { "objects", FSYNC_COMPONENTS_OBJECTS }, >> >> >> > + { "default", FSYNC_COMPONENTS_DEFAULT }, >> >> >> > + { "all", FSYNC_COMPONENTS_ALL }, >> >> >> > +}; >> >> >> > + >> >> >> > +static enum fsync_component parse_fsync_components(const char *var, const char *string) >> >> >> > +{ >> >> >> > + enum fsync_component output = 0; >> >> >> > + >> >> >> > + if (!strcmp(string, "none")) >> >> >> > + return output; >> >> >> > + >> >> >> > + while (string) { >> >> >> > + int i; >> >> >> > + size_t len; >> >> >> > + const char *ep; >> >> >> > + int negated = 0; >> >> >> > + int found = 0; >> >> >> > + >> >> >> > + string = string + strspn(string, ", \t\n\r"); >> >> >> >> >> >> Aside from the "use a list" isn't this hardcoding some windows-specific >> >> >> assumptions with \n\r? Maybe not... >> >> > >> >> > I shamelessly stole this code from parse_whitespace_rule. I thought >> >> > about making a helper to be called by both functions, but the amount >> >> > of state going into and out of the wrapper via arguments was >> >> > substantial and seemed to negate the benefit of deduplication. >> >> >> >> FWIW string_list_split() is easier to work with in those cases, or at >> >> least I think so... >> > >> > This code runs at startup for a variable that may be present on some >> > installations. The nice property of the current patch's code is that >> > it's already a well-tested pattern that doesn't do any allocations as >> > it's working, unlike string_list_split(). >> >> Multi-value config would also get you fewer allocations :) >> >> Anyway, I mainly meant to point out that for stuff like this it's fine >> to optimize it for ease rather than micro-optimize allocations. Those >> really aren't a bottleneck on this scale. >> >> Even in that case there's string_list_split_in_place(), which can be a >> bit nicer than manual C-string fiddling. >> > > Am I allowed to change the config value string in place? The > core.whitespace code is careful not to modify the string. I kind of > like the parse_ws_error_highlight code a little better now that I've > seen it, but I think the current code is fine too. I don't remember offhand if that's safe, probably not. So you'll need a copy here. >> > I hope you know that I appreciate your review feedback, even though >> > I'm pushing back on most of it so far this round. I'll be sending v3 >> > to the list soon after giving it another look over. >> >> Sure, no worries. Just hoping to help. If you go for something different >> etc. that's fine. Just hoping to bridge the gap in some knowledge / >> offer potentially interesting suggestions (some of which may be dead >> ends, like the config glob thing...). ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v2 2/3] core.fsync: introduce granular fsync control 2022-01-19 14:52 ` Ævar Arnfjörð Bjarmason @ 2022-01-28 1:28 ` Neeraj Singh 0 siblings, 0 replies; 122+ messages in thread From: Neeraj Singh @ 2022-01-28 1:28 UTC (permalink / raw) To: Ævar Arnfjörð Bjarmason Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Patrick Steinhardt, Neeraj Singh On Wed, Jan 19, 2022 at 10:27 AM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote: > > > On Wed, Dec 08 2021, Neeraj Singh wrote: > > [Sorry about the late reply, and thanks for the downthread prodding] I also apologize for the late reply here. I've been dealing with a hospitalized parent this week. > > > On Wed, Dec 8, 2021 at 8:46 PM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote: > >> > >> > >> On Wed, Dec 08 2021, Neeraj Singh wrote: > >> > >> > On Wed, Dec 8, 2021 at 2:17 AM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote: > >> >> > >> >> > >> >> On Tue, Dec 07 2021, Neeraj Singh wrote: > >> >> > >> >> > On Tue, Dec 7, 2021 at 5:01 AM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote: > >> >> >> > >> >> >> > >> >> >> On Tue, Dec 07 2021, Neeraj Singh via GitGitGadget wrote: > >> >> >> > >> >> >> > From: Neeraj Singh <neerajsi@microsoft.com> > >> >> >> > > >> >> >> > This commit introduces the `core.fsync` configuration > >> >> >> > knob which can be used to control how components of the > >> >> >> > repository are made durable on disk. > >> >> >> > > >> >> >> > This setting allows future extensibility of the list of > >> >> >> > syncable components: > >> >> >> > * We issue a warning rather than an error for unrecognized > >> >> >> > components, so new configs can be used with old Git versions. > >> >> >> > >> >> >> Looks good! > >> >> >> > >> >> >> > * We support negation, so users can choose one of the default > >> >> >> > aggregate options and then remove components that they don't > >> >> >> > want. The user would then harden any new components added in > >> >> >> > a Git version update. > >> >> >> > >> >> >> I think this config schema makes sense, but just a (I think important) > >> >> >> comment on the "how" not "what" of it. It's really much better to define > >> >> >> config as: > >> >> >> > >> >> >> [some] > >> >> >> key = value > >> >> >> key = value2 > >> >> >> > >> >> >> Than: > >> >> >> > >> >> >> [some] > >> >> >> key = value,value2 > >> >> >> > >> >> >> The reason is that "git config" has good support for working with > >> >> >> multi-valued stuff, so you can do e.g.: > >> >> >> > >> >> >> git config --get-all -z some.key > >> >> >> > >> >> >> And you can easily (re)set such config e.g. with --replace-all etc., but > >> >> >> for comma-delimited you (and users) need to do all that work themselves. > >> >> >> > >> >> >> Similarly instead of: > >> >> >> > >> >> >> some.key = want-this > >> >> >> some.key = -not-this > >> >> >> some.key = but-want-this > >> >> >> > >> >> >> I think it's better to just have two lists, one inclusive another > >> >> >> exclusive. E.g. see "log.decorate" and "log.excludeDecoration", > >> >> >> "transfer.hideRefs" > >> >> >> > >> >> >> Which would mean: > >> >> >> > >> >> >> core.fsync = want-this > >> >> >> core.fsyncExcludes = -not-this > >> >> >> > >> >> >> For some value of "fsyncExcludes", maybe "noFsync"? Anyway, just a > >> >> >> suggestion on making this easier for users & the implementation. > >> >> >> > >> >> > > >> >> > Maybe there's some way to handle this I'm unaware of, but a > >> >> > disadvantage of your multi-valued config proposal is that it's harder, > >> >> > for example, for a per-repo config store to reasonably override a > >> >> > per-user config store. With the configuration scheme as-is, I can > >> >> > have a per-user setting like `core.fsync=all` which covers my typical > >> >> > repos, but then have a maintainer repo with a private setting of > >> >> > `core.fsync=none` to speed up cases where I'm mostly working with > >> >> > other people's changes that are backed up in email or server-side > >> >> > repos. The latter setting conveniently overrides the former setting > >> >> > in all aspects. > >> >> > >> >> Even if you turn just your comma-delimited proposal into a list proposal > >> >> can't we just say that the last one wins? Then it won't matter what cmae > >> >> before, you'd specify "core.fsync=none" in your local .git/config. > >> >> > >> >> But this is also a general issue with a bunch of things in git's config > >> >> space. I'd rather see us use the list-based values and just come up with > >> >> some general way to reset them that works for all keys, rather than > >> >> regretting having comma-delimited values that'll be harder to work with > >> >> & parse, which will be a legacy wart if/when we come up with a way to > >> >> say "reset all previous settings". > >> >> > >> >> > Also, with the core.fsync and core.fsyncExcludes, how would you spell > >> >> > "don't sync anything"? Would you still have the aggregate options.? > >> >> > >> >> core.fsyncExcludes = * > >> >> > >> >> I.e. the same as the core.fsync=none above, anyway I still like the > >> >> wildcard thing below a bit more... > >> > > >> > I'm not going to take this feedback unless there are additional votes > >> > from the Git community in this direction. I make the claim that > >> > single-valued comma-separated config lists are easier to work with in > >> > the existing Git infrastructure. > >> > >> Easier in what sense? I showed examples of how "git-config" trivially > >> works with multi-valued config, but for comma-delimited you'll need to > >> write your own shellscript boilerplate around simple things like adding > >> values, removing existing ones etc. > >> > >> I.e. you can use --add, --unset, --unset-all, --get, --get-all etc. > >> > > > > I see what you're saying for cases where someone would want to set a > > core.fsync setting that's derived from the user's current config. But > > I'm guessing that the dominant use case is someone setting a new fsync > > configuration that achieves some atomic goal with respect to a given > > repository. Like "this is a throwaway, so sync nothing" or "this is > > really important, so sync all objects and refs and the index". > > Whether it's multi-value or comma-separated you could do: > > -c core.fsync=[none|false] > > To sync nothing, i.e. if we see "none/false" it doesn't matter if we saw > core.fsync=loose-object or whatever before, it would override it, ditto > for "all". > > >> > parsing code for the core.whitespace variable and users are used to > >> > this syntax there. There are several other comma-separated lists in > >> > the config space, so this construct has precedence and will be with > >> > Git for some time. > >> > >> That's not really an argument either way for why we'd pick X over Y for > >> something new. We've got some comma-delimited, some multi-value (I'm > >> fairly sure we have more multi-value). > >> > > > > My main point here is that there's precedent for patch's current exact > > schema for a config in the same core config leaf of the Documentation. > > It seems from your comments that we'd have to invent and document some > > new convention for "reset" of a multi-valued config. So you're > > suggesting that I solve an extra set of problems to get this change > > in. Just want to remind you that my personal itch to scratch is to > > get the underlying mechanism in so that git-for-windows can set its > > default setting to batch mode. I'm not expecting many users to > > actually configure this setting to any non-default value. > > Me neither. I think people will most likely set this once in > ~/.gitconfig or /etc/gitconfig. > > We have some config keys that are multi-value and either comma-separated > or space-separated, e.g. core.alternateRefsPrefixes > > Then we have e.g. blame.ignoreRevsFile which is multi-value, and further > has the convention that setting it to an empty value clears the > list. which would scratch the "override existing" itch. > > format.notes, versionsort.suffix, transfer.hideRefs, branch.<name>.merge > are exmples of existing multi-value config. > Thanks for the examples. I can see the benefit of mutli-value for most of those settings, but for versionsort.suffix, I'd personally have wanted a comma-separated list. > >> > Also, fsync configurations aren't composable like > >> > some other configurations may be. It makes sense to have a holistic > >> > singular fsync configuration, which is best represented by a single > >> > variable. > >> > >> What's a "variable" here? We call these "keys", you can have a > >> single-value key like user.name that you get with --get, or a > >> multi-value key like say branch.<name>.merge or push.pushOption that > >> you'd get with --get-all. > > > > Yeah, I meant "key". I conflated the config key with the underlying > > global variable in git. > > *nod* > > >> I think you may be referring to either not wanting these to be > >> "inherited" (which is not really a think we do for anything else in > >> config), or lacking the ability to "reset". > >> > >> For the latter if that's absolutely needed we could just use the same > >> trick as "diff.wsErrorHighlight" uses of making an empty value "reset" > >> the list, and you'd get better "git config" support for editing it. > >> > > > > My reading of the code is that diff.wsErrorHighlight is a comma > > separated list and not a multi-valued config. Actually I haven't yet > > found an existing multi-valued config (not sure how to grep for it). > > Yes, I think I conflated it with one of the ones above when I wrote > this. > > >> >> >> > This also supports the common request of doing absolutely no > >> >> >> > fysncing with the `core.fsync=none` value, which is expected > >> >> >> > to make the test suite faster. > >> >> >> > >> >> >> Let's just use the git_parse_maybe_bool() or git_parse_maybe_bool_text() > >> >> >> so we'll accept "false", "off", "no" like most other such config? > >> >> > > >> >> > Junio's previous feedback when discussing batch mode [1] was to offer > >> >> > less flexibility when parsing new values of these configuration > >> >> > options. I agree with his statement that "making machine-readable > >> >> > tokens be spelled in different ways is a 'disease'." I'd like to > >> >> > leave this as-is so that the documentation can clearly state the exact > >> >> > set of allowable values. > >> >> > > >> >> > [1] https://lore.kernel.org/git/xmqqr1dqzyl7.fsf@gitster.g/ > >> >> > >> >> I think he's talking about batch, Batch, BATCH, bAtCh etc. there. But > >> >> the "maybe bool" is a stanard pattern we use. > >> >> > >> >> I don't think we'd call one of these 0, off, no or false etc. to avoid > >> >> confusion, so then you can use git_parse_maybe_...() > >> > > >> > I don't see the advantage of having multiple ways of specifying > >> > "none". The user can read the doc and know exactly what to write. If > >> > they write something unallowable, they get a clear warning and they > >> > can read the doc again to figure out what to write. This isn't a > >> > boolean options at all, so why should we entertain bool-like ways of > >> > spelling it? > >> > >> It's not boolean, it's multi-value and one of the values includes a true > >> or false boolean value. You just spell it "none". > >> > >> I think both this and your comment above suggest that you think there's > >> no point in this because you haven't interacted with/used "git config" > >> as a command line or API mechanism, but have just hand-crafted config > >> files. > >> > >> That's fair enough, but there's a lot of tooling that benefits from the > >> latter. > > > > My batch mode perf tests (on github, not yet submitted to the list) > > use `git -c core.fsync=<foo>` to set up a per-process config. I > > haven't used the `git config` writing support in a while, so I haven't > > deeply thought about that. However, I favor simplifying the use case > > of "atomically" setting a new holistic core.fsync value versus the use > > case of deriving a new core.fsync value from the preexisting value. > > If you implement it like blame.ignoreRevsFile you can have your cake and > eat it too, i.e.: > > -c core.fsync= core.fsync=loose-object > > ensures only loose objects are synced, as with your single-value config, > but I'd think what you'd be more likely to actually mean would be: > > # With "core.fsync=pack" set in ~/.gitconfig > -c core.fsync=loose-object > > I.e. that the common case is "I want this to be synced here", not that > you'd like to declare sync policy from scratch every time. > > In any case, on this general topic my main point is that the > git-config(1) command has pretty integration for multi-value if you do > it that way, and not for comma-delimited. I.e. you get --add, --unset, > --unset-all, --get, --get-all etc. > > So I think for anything new it makes sense to lean into that, I think > most of these existing comma-delimited ones are ones we'd do differently > today on reflection. > > And if you suppor the "empty resets" like blame.ignoreRevsFile it seems > to me you'll have your cake & eat it too. > I really don't like the multiple `-c core.fsync=` clauses. If I want to do packs and loose-objects as a single config, I'd have to do: * multi-value: `git -c core.fsync= -c core.fsync=pack -c core.fsync=loose-object` * comma-sep `git -c core.fsync=pack,loose-object` Multi-valued configs are stateful and more verbose to configure. Last-one-wins with comma-separated values has an advantage for achieving a desired final configuration without regard for the previous configuration, which is the way I expect the feature to be used. > >> E.g.: > >> > >> $ git -c core.fsync=off config --type=bool core.fsync > >> false > >> $ git -c core.fsync=blah config --type=bool core.fsync > >> fatal: bad boolean config value 'blah' for 'core.fsync' > >> > >> Here we can get 'git config' to normalize what you call 'none', and you > >> can tell via exit codes/normalization if it's "false". But if you invent > >> a new term for "false" you can't do that as easily. > >> > >> We have various historical keys that take odd exceptions to that, > >> e.g. "never", but unless we have a good reason to let's not invent more > >> exceptions. > >> > >> >> >> > Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> > >> >> >> > --- > >> >> >> > Documentation/config/core.txt | 27 +++++++++---- > >> >> >> > builtin/fast-import.c | 2 +- > >> >> >> > builtin/index-pack.c | 4 +- > >> >> >> > builtin/pack-objects.c | 8 ++-- > >> >> >> > bulk-checkin.c | 5 ++- > >> >> >> > cache.h | 39 +++++++++++++++++- > >> >> >> > commit-graph.c | 3 +- > >> >> >> > config.c | 76 ++++++++++++++++++++++++++++++++++- > >> >> >> > csum-file.c | 5 ++- > >> >> >> > csum-file.h | 3 +- > >> >> >> > environment.c | 1 + > >> >> >> > midx.c | 3 +- > >> >> >> > object-file.c | 3 +- > >> >> >> > pack-bitmap-write.c | 3 +- > >> >> >> > pack-write.c | 13 +++--- > >> >> >> > read-cache.c | 2 +- > >> >> >> > 16 files changed, 164 insertions(+), 33 deletions(-) > >> >> >> > > >> >> >> > diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt > >> >> >> > index dbb134f7136..4f1747ec871 100644 > >> >> >> > --- a/Documentation/config/core.txt > >> >> >> > +++ b/Documentation/config/core.txt > >> >> >> > @@ -547,6 +547,25 @@ core.whitespace:: > >> >> >> > is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent` > >> >> >> > errors. The default tab width is 8. Allowed values are 1 to 63. > >> >> >> > > >> >> >> > +core.fsync:: > >> >> >> > + A comma-separated list of parts of the repository which should be > >> >> >> > + hardened via the core.fsyncMethod when created or modified. You can > >> >> >> > + disable hardening of any component by prefixing it with a '-'. Later > >> >> >> > + items take precedence over earlier ones in the list. For example, > >> >> >> > + `core.fsync=all,-pack-metadata` means "harden everything except pack > >> >> >> > + metadata." Items that are not hardened may be lost in the event of an > >> >> >> > + unclean system shutdown. > >> >> >> > ++ > >> >> >> > +* `none` disables fsync completely. This must be specified alone. > >> >> >> > +* `loose-object` hardens objects added to the repo in loose-object form. > >> >> >> > +* `pack` hardens objects added to the repo in packfile form. > >> >> >> > +* `pack-metadata` hardens packfile bitmaps and indexes. > >> >> >> > +* `commit-graph` hardens the commit graph file. > >> >> >> > +* `objects` is an aggregate option that includes `loose-objects`, `pack`, > >> >> >> > + `pack-metadata`, and `commit-graph`. > >> >> >> > +* `default` is an aggregate option that is equivalent to `objects,-loose-object` > >> >> >> > +* `all` is an aggregate option that syncs all individual components above. > >> >> >> > + > >> >> >> > >> >> >> It's probably a *bit* more work to set up, but I wonder if this wouldn't > >> >> >> be simpler if we just said (and this is partially going against what I > >> >> >> noted above): > >> >> >> > >> >> >> == BEGIN DOC > >> >> >> > >> >> >> core.fsync is a multi-value config variable where each item is a > >> >> >> pathspec that'll get matched the same way as 'git-ls-files' et al. > >> >> >> > >> >> >> When we sync pretend that a path like .git/objects/de/adbeef... is > >> >> >> relative to the top-level of the git > >> >> >> directory. E.g. "objects/de/adbeaf.." or "objects/pack/...". > >> >> >> > >> >> >> You can then supply a list of wildcards and exclusions to configure > >> >> >> syncing. or "false", "off" etc. to turn it off. These are synonymous > >> >> >> with: > >> >> >> > >> >> >> ; same as "false" > >> >> >> core.fsync = ":!*" > >> >> >> > >> >> >> Or: > >> >> >> > >> >> >> ; same as "true" > >> >> >> core.fsync = "*" > >> >> >> > >> >> >> Or, to selectively sync some things and not others: > >> >> >> > >> >> >> ;; Sync objects, but not "info" > >> >> >> core.fsync = ":!objects/info/**" > >> >> >> core.fsync = "objects/**" > >> >> >> > >> >> >> See gitrepository-layout(5) for details about what sort of paths you > >> >> >> might be expected to match. Not all paths listed there will go through > >> >> >> this mechanism (e.g. currently objects do, but nothing to do with config > >> >> >> does). > >> >> >> > >> >> >> We can and will match this against "fake paths", e.g. when writing out > >> >> >> packs we may match against just the string "objects/pack", we're not > >> >> >> going to re-check if every packfile we're writing matches your globs, > >> >> >> ditto for loose objects. Be reasonable! > >> >> >> > >> >> >> This metharism is intended as a shorthand that provides some flexibility > >> >> >> when fsyncing, while not forcing git to come up with labels for all > >> >> >> paths the git dir, or to support crazyness like "objects/de/adbeef*" > >> >> >> > >> >> >> More paths may be added or removed in the future, and we make no > >> >> >> promises that we won't move things around, so if in doubt use > >> >> >> e.g. "true" or a wide pattern match like "objects/**". When in doubt > >> >> >> stick to the golden path of examples provided in this documentation. > >> >> >> > >> >> >> == END DOC > >> >> >> > >> >> >> > >> >> >> It's a tad more complex to set up, but I wonder if that isn't worth > >> >> >> it. It nicely gets around any current and future issues of deciding what > >> >> >> labels such as "loose-object" etc. to pick, as well as slotting into an > >> >> >> existing method of doing exclude/include lists. > >> >> >> > >> >> > > >> >> > I think this proposal is a lot of complexity to avoid coming up with a > >> >> > new name for syncable things as they are added to Git. A path based > >> >> > mechanism makes it hard to document for the (advanced) user what the > >> >> > full set of things is and how it might change from release to release. > >> >> > I think the current core.fsync scheme is a bit easier to understand, > >> >> > query, and extend. > >> >> > >> >> We document it in gitrepository-layout(5). Yeah it has some > >> >> disadvantages, but one advantage is that you could make the > >> >> composability easy. I.e. if last exclude wins then a setting of: > >> >> > >> >> core.fsync = ":!*" > >> >> core.fsync = "objects/**" > >> >> > >> >> Would reset all previous matches & only match objects/**. > >> >> > >> > > >> > The value of changing this is predicated on taking your previous > >> > multi-valued config proposal, which I'm still not at all convinced > >> > about. > >> > >> They're orthagonal. I.e. you get benefits from multi-value with or > >> without this globbing mechanism. > >> > >> In any case, I don't feel strongly about/am really advocating this > >> globbing mechanism. I just wondered if it wouldn't make things simpler > >> since it would sidestep the need to create any sort of categories for > >> subsets of gitrepository-layout(5), but maybe not... > >> > >> > The schema in the current (v1-v2) version of the patch already > >> > includes an example of extending the list of syncable things, and > >> > Patrick Steinhardt made it clear that he feels comfortable adding > >> > 'refs' to the same schema in a future change. > >> > > >> > I'll also emphasize that we're talking about a non-functional, > >> > relatively corner-case behavioral configuration. These values don't > >> > change how git's interface behaves except when the system crashes > >> > during a git command or shortly after one completes. > >> > >> That may be something some OS's promise, but it's not something fsync() > >> or POSIX promises. I.e. you might write a ref, but unless you fsync and > >> the relevant dir entries another process might not see the update, crash > >> or not. > >> > > > > I haven't seen any indication that POSIX requires an fsync for > > visiblity within a running system. I looked at the doc for open, > > write, and fsync, and saw no indication that it's posix compliant to > > require an fsync for visibility. I think an OS that required fsync > > for cross-process visiblity would fail to run Git for a myriad of > > other reasons and would likely lose all its users. I'm curious where > > you've seen documentation that allows such unhelpful behavior? > > There's multiple unrelated and related things in this area. One is a > case where you'll e.g. write a file "foo" using stdio, spawn a program > to work on it in the same program, but it might not see it at all, or > see empty content, the latter being because you haven't flushed your I/O > buffers (which you can do via fsync()). > For stdio you need to use fflush(3), which just flushes the C runtime's internal buffers. You need to do the following to do a full durable write using stdio: ``` FILE *fp; ... fflush(fp); fsync(fileno(fp)) ``` > The former is that on *nix systems you're generally only guaranteed to > write to a fd, but not to have the associated metadata be synced for > you. > > That is spelled out e.g. in the DESCRIPTION section of linux's fsync() > manpage: https://man7.org/linux/man-pages/man2/fdatasync.2.html > > I don't know how much you follow non-Windows FS development, but there > was also a very well known "incident" early in ext4 where it leaned into > some permissive-by-POSIX behavior that caused data loss in practice on > buggy programs that didn't correctly use fsync() , since various tooling > had come to expect the stricter behavior of ext3: > https://lwn.net/Articles/328363/ > > That was explicitly in the area of fs metadata being discussed here. > > Generally you can expect your VFS layer to be forgiving when it comes to > IPC, but even that is out the window when it comes to networked > filesystems, e.g. a shared git repository hosted on NFS. > Everything in the fsync(2) DESCRIPTION section is about what data and metadata reaches the disk (versus just being cached in-memory). I've become a bit familiar with the ext3 vs ext4 (and delayed alloc) behavior while researching this feature. These behaviors are all around the durability you get in the case of kernel crash, power-failure, or other forms of dirty dismount. NFS is a complex story. I'm not intimately familiar with its particular pitfalls, but from looking at the Linux NFS faq (http://nfs.sourceforge.net/), it appears that a given single NFS client will remain coherent with itself. Multiple NFS clients accessing a single Git repo concurrently are probably going to see some inconsistency. In that kind of case, fsync would help, perhaps, since it would force NFS clients to issue a COMMIT command to the server. > >> That's an aside from these config design questions, and I think > >> most/(all?) OS's anyone cares about these days tend to make that > >> implicit promise as part of their VFS behavior, but we should probably > >> design such an interface to fsync() with such pedantic portability in > >> mind. > > > > Why? To be rude to such a hypothetical system, if a system were so > > insanely designed, it would be nuts to support it. > > Because we know that right now the system calls we're invoking aren't > guaranteed to store data persistently to disk portably, although they do > so in practice on most modern OS's. > > We're portably to a lot of platforms, and also need to keep e.g. NFS in > mind, so being able to ask for a pedantic mode when you care about data > retention at the cost of performance would be nice. > > And because the fsync config mode you're proposing is thoroughly > non-standard, but is known to me much faster by leaning into known > attributes of specific FS's on specific OS's, if we're not running on > those it would be sensible to fall back to a stricter mode of > operation. E.g. syncing all 100 loose objects we just wrote, not just > the last one. > > >> > While you may not personally love the proposed configuration > >> > interface, I'd want your view on some questions: > >> > 1. Is it easy for the (advanced) user to set a configuration? > >> > 2. Is it easy for the (advanced) user to see what was configured? > >> > 3. Is it easy for the Git community to build on this as we want to add > >> > things to the list of things to sync? > >> > a) Is there a good best practice configuration so that people can > >> > avoid losing integrity for new stuff that they are intending to sync. > >> > b) If someone has a custom configuration, can that custom > >> > configuration do something reasonable as they upgrade versions of Git? > >> > ** In response to this question, I might see some value > >> > in adding a 'derived-metadata' aggregate that can be disabled so that > >> > a custom configuration can exclude those as they change version to > >> > version. > >> > c) Is it too much maintenance overhead to consider how to present > >> > this configuration knob for any new hashfile or other datafile in the > >> > git repo? > >> > 4. Is there a good path forward to change the default syncable set, > >> > both in git-for-windows and in Git for other platforms? > >> > >> I'm not really sure this globbing this is a good idea, as noted above > >> just a suggestion etc. > >> > >> As noted there it just gets you out of the business of re-defining > >> gitrepository-layout(5), and assuming too much in advance about certain > >> use-cases. > >> > >> E.g. even "refs" might be too broad for some. I don't tend to be I/O > >> limited, but I could see how someone who would be would care about > >> refs/heads but not refs/remotes, or want to exclude logs/* but not the > >> refs updates themselves etc. > > > > This use-case is interesting (distinguishing remote refs from local > > refs). I think the difficulty of verifying (for even an advanced > > user) that the right fsyncing is actually happening still puts me on > > the side of having a carefully curated and documented set of syncable > > things rather than a file-path-based mechanism. > > > > Is this meaningful in the presumably nearby future world of the refsdb > > backend? Is that somehow split by remote versus local? > > There is the upcoming "reftable" work, but that's probably 2-3 years out > at the earliest for series production workloads in git.git. > > >> >> >> > diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c > >> >> >> > index 857be7826f3..916c55d6ce9 100644 > >> >> >> > --- a/builtin/pack-objects.c > >> >> >> > +++ b/builtin/pack-objects.c > >> >> >> > @@ -1204,11 +1204,13 @@ static void write_pack_file(void) > >> >> >> > * If so, rewrite it like in fast-import > >> >> >> > */ > >> >> >> > if (pack_to_stdout) { > >> >> >> > - finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_CLOSE); > >> >> >> > + finalize_hashfile(f, hash, FSYNC_COMPONENT_NONE, > >> >> >> > + CSUM_HASH_IN_STREAM | CSUM_CLOSE); > >> >> >> > >> >> >> Not really related to this per-se, but since you're touching the API > >> >> >> everything goes through I wonder if callers should just always try to > >> >> >> fsync, and we can just catch EROFS and EINVAL in the wrapper if someone > >> >> >> tries to flush stdout, or catch the fd at that lower level. > >> >> >> > >> >> >> Or maybe there's a good reason for this... > >> >> > > >> >> > It's platform dependent, but I'd expect fsync would do something for > >> >> > pipes or stdout redirected to a file. In these cases we really don't > >> >> > want to fsync since we have no idea what we're talking to and we're > >> >> > potentially worsening performance for probably no benefit. > >> >> > >> >> Yeah maybe we should just leave it be. > >> >> > >> >> I'd think the C library returning EINVAL would be a trivial performance > >> >> cost though. > >> >> > >> >> It just seemed odd to hardcode assumptions about what can and can't be > >> >> synced when the POSIX defined function will also tell us that. > >> >> > >> > > >> > Redirecting stdout to a file seems like a common usage for this > >> > command. That would definitely be fsyncable, but Git has no idea what > >> > its proper category is since there's no way to know the purpose or > >> > lifetime of the packfile. I'm going to leave this be, because I'd > >> > posit that "can it be fsynced?" is not the same as "should it be > >> > fsynced?". The latter question can't be answered for stdout. > >> > >> As noted this was just an aside, and I don't even know if any OS would > >> do anything meaningful with an fsync() of such a FD anyway. > >> > > > > The underlying fsync primitive does have a meaning on Windows for > > pipes, but it's certainly not what Git would want to do. Also if > > stdout is redirected to a file, I'm pretty sure that UNIX OSes would > > respect the fsync call. However it's not meaningful in the sense of > > the git repository, since we don't know what the packfile is or why it > > was created. > > I suggested that because I think it's probably nonsensical, but it's > nonsense that POSIX seems to explicitly tell us that it'll handle > (probably by silently doing nothing). So in terms of our interface we > could lean into that and avoid our own special-casing. > > >> I just don't see why we wouldn't say: > >> > >> 1. We're syncing this category of thing > >> 2. Try it > >> 3. If fsync returns "can't fsync that sort of thing" move on > >> > >> As opposed to trying to shortcut #3 by doing the detection ourselves. > >> > >> I.e. maybe there was a good reason, but it seemed to be some easy > >> potential win for more simplification, since you were re-doing and > >> simplifying some of the interface anyway... > > > > We're trying to be deliberate about what we're fsyncing. Fsyncing an > > unknown file created by the packfile code doesn't move us in that > > direction. In your taxonomy we don't know (1), "what is this category > > of thing?" Sure it's got the packfile format, but is not known to be > > an actual packfile that's part of the repository. > > We know it's a fd, isn't that sufficient? In any case, I'm fine with > also keeping it as is, I don't mean to split hairs here. > > It just stuck out as an odd part of the interface, why treat some fd's > specially, instead of just throwing it all at the OS. Presumably the > first thing the OS will do is figure out if it's a syncable fd or not, > and act appropriately. > I'll put the following comment in pack-objects.c: /* * We never fsync when writing to stdout since we may * not be writing to a specific file. For instance, the * upload-pack code passes a pipe here. Calling fsync * on a pipe results in unnecessary synchronization with * the reader on some platforms. */ > >> >> > >> >> >> > [...] > >> >> >> > +/* > >> >> >> > + * These values are used to help identify parts of a repository to fsync. > >> >> >> > + * FSYNC_COMPONENT_NONE identifies data that will not be a persistent part of the > >> >> >> > + * repository and so shouldn't be fsynced. > >> >> >> > + */ > >> >> >> > +enum fsync_component { > >> >> >> > + FSYNC_COMPONENT_NONE = 0, > >> >> >> > >> >> >> I haven't read ahead much but in most other such cases we don't define > >> >> >> the "= 0", just start at 1<<0, then check the flags elsewhere... > >> >> >> > >> >> >> > +static const struct fsync_component_entry { > >> >> >> > + const char *name; > >> >> >> > + enum fsync_component component_bits; > >> >> >> > +} fsync_component_table[] = { > >> >> >> > + { "loose-object", FSYNC_COMPONENT_LOOSE_OBJECT }, > >> >> >> > + { "pack", FSYNC_COMPONENT_PACK }, > >> >> >> > + { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA }, > >> >> >> > + { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, > >> >> >> > + { "objects", FSYNC_COMPONENTS_OBJECTS }, > >> >> >> > + { "default", FSYNC_COMPONENTS_DEFAULT }, > >> >> >> > + { "all", FSYNC_COMPONENTS_ALL }, > >> >> >> > +}; > >> >> >> > + > >> >> >> > +static enum fsync_component parse_fsync_components(const char *var, const char *string) > >> >> >> > +{ > >> >> >> > + enum fsync_component output = 0; > >> >> >> > + > >> >> >> > + if (!strcmp(string, "none")) > >> >> >> > + return output; > >> >> >> > + > >> >> >> > + while (string) { > >> >> >> > + int i; > >> >> >> > + size_t len; > >> >> >> > + const char *ep; > >> >> >> > + int negated = 0; > >> >> >> > + int found = 0; > >> >> >> > + > >> >> >> > + string = string + strspn(string, ", \t\n\r"); > >> >> >> > >> >> >> Aside from the "use a list" isn't this hardcoding some windows-specific > >> >> >> assumptions with \n\r? Maybe not... > >> >> > > >> >> > I shamelessly stole this code from parse_whitespace_rule. I thought > >> >> > about making a helper to be called by both functions, but the amount > >> >> > of state going into and out of the wrapper via arguments was > >> >> > substantial and seemed to negate the benefit of deduplication. > >> >> > >> >> FWIW string_list_split() is easier to work with in those cases, or at > >> >> least I think so... > >> > > >> > This code runs at startup for a variable that may be present on some > >> > installations. The nice property of the current patch's code is that > >> > it's already a well-tested pattern that doesn't do any allocations as > >> > it's working, unlike string_list_split(). > >> > >> Multi-value config would also get you fewer allocations :) > >> > >> Anyway, I mainly meant to point out that for stuff like this it's fine > >> to optimize it for ease rather than micro-optimize allocations. Those > >> really aren't a bottleneck on this scale. > >> > >> Even in that case there's string_list_split_in_place(), which can be a > >> bit nicer than manual C-string fiddling. > >> > > > > Am I allowed to change the config value string in place? The > > core.whitespace code is careful not to modify the string. I kind of > > like the parse_ws_error_highlight code a little better now that I've > > seen it, but I think the current code is fine too. > > I don't remember offhand if that's safe, probably not. So you'll need a > copy here. > > >> > I hope you know that I appreciate your review feedback, even though > >> > I'm pushing back on most of it so far this round. I'll be sending v3 > >> > to the list soon after giving it another look over. > >> > >> Sure, no worries. Just hoping to help. If you go for something different > >> etc. that's fine. Just hoping to bridge the gap in some knowledge / > >> offer potentially interesting suggestions (some of which may be dead > >> ends, like the config glob thing...). Thanks again for the review, I'll send an updated PR soon. Thanks, Neeraj ^ permalink raw reply [flat|nested] 122+ messages in thread
* [PATCH v2 3/3] core.fsync: new option to harden the index 2021-12-07 2:46 ` [PATCH v2 0/3] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget 2021-12-07 2:46 ` [PATCH v2 1/3] core.fsyncmethod: add writeout-only mode Neeraj Singh via GitGitGadget 2021-12-07 2:46 ` [PATCH v2 2/3] core.fsync: introduce granular fsync control Neeraj Singh via GitGitGadget @ 2021-12-07 2:46 ` Neeraj Singh via GitGitGadget 2021-12-07 11:56 ` [PATCH v2 0/3] A design for future-proofing fsync() configuration Patrick Steinhardt 2021-12-09 0:57 ` [PATCH v3 0/4] " Neeraj K. Singh via GitGitGadget 4 siblings, 0 replies; 122+ messages in thread From: Neeraj Singh via GitGitGadget @ 2021-12-07 2:46 UTC (permalink / raw) To: git Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps, Neeraj K. Singh, Neeraj Singh From: Neeraj Singh <neerajsi@microsoft.com> This commit introduces the new ability for the user to harden the index. In the event of a system crash, the index must be durable for the user to actually find a file that has been added to the repo and then deleted from the working tree. We use the presence of the COMMIT_LOCK flag and absence of the alternate_index_output as a proxy for determining whether we're updating the persistent index of the repo or some temporary index. We don't sync these temporary indexes. Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> --- Documentation/config/core.txt | 1 + cache.h | 4 +++- config.c | 1 + read-cache.c | 19 +++++++++++++------ 4 files changed, 18 insertions(+), 7 deletions(-) diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt index 4f1747ec871..8e5b7a795ab 100644 --- a/Documentation/config/core.txt +++ b/Documentation/config/core.txt @@ -561,6 +561,7 @@ core.fsync:: * `pack` hardens objects added to the repo in packfile form. * `pack-metadata` hardens packfile bitmaps and indexes. * `commit-graph` hardens the commit graph file. +* `index` hardens the index when it is modified. * `objects` is an aggregate option that includes `loose-objects`, `pack`, `pack-metadata`, and `commit-graph`. * `default` is an aggregate option that is equivalent to `objects,-loose-object` diff --git a/cache.h b/cache.h index d83fbaf2619..4dc26d7b2c9 100644 --- a/cache.h +++ b/cache.h @@ -996,6 +996,7 @@ enum fsync_component { FSYNC_COMPONENT_PACK = 1 << 1, FSYNC_COMPONENT_PACK_METADATA = 1 << 2, FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3, + FSYNC_COMPONENT_INDEX = 1 << 4, }; #define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \ @@ -1010,7 +1011,8 @@ enum fsync_component { #define FSYNC_COMPONENTS_ALL (FSYNC_COMPONENT_LOOSE_OBJECT | \ FSYNC_COMPONENT_PACK | \ FSYNC_COMPONENT_PACK_METADATA | \ - FSYNC_COMPONENT_COMMIT_GRAPH) + FSYNC_COMPONENT_COMMIT_GRAPH | \ + FSYNC_COMPONENT_INDEX) /* diff --git a/config.c b/config.c index 29c867aab03..17039fa9c10 100644 --- a/config.c +++ b/config.c @@ -1221,6 +1221,7 @@ static const struct fsync_component_entry { { "pack", FSYNC_COMPONENT_PACK }, { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA }, { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, + { "index", FSYNC_COMPONENT_INDEX }, { "objects", FSYNC_COMPONENTS_OBJECTS }, { "default", FSYNC_COMPONENTS_DEFAULT }, { "all", FSYNC_COMPONENTS_ALL }, diff --git a/read-cache.c b/read-cache.c index f3539681f49..783cb3ea5db 100644 --- a/read-cache.c +++ b/read-cache.c @@ -2816,7 +2816,7 @@ static int record_ieot(void) * rely on it. */ static int do_write_index(struct index_state *istate, struct tempfile *tempfile, - int strip_extensions) + int strip_extensions, unsigned flags) { uint64_t start = getnanotime(); struct hashfile *f; @@ -2830,6 +2830,7 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile, struct strbuf previous_name_buf = STRBUF_INIT, *previous_name; int drop_cache_tree = istate->drop_cache_tree; off_t offset; + int csum_fsync_flag; int ieot_entries = 1; struct index_entry_offset_table *ieot = NULL; int nr, nr_threads; @@ -3060,7 +3061,13 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile, return -1; } - finalize_hashfile(f, istate->oid.hash, FSYNC_COMPONENT_NONE, CSUM_HASH_IN_STREAM); + csum_fsync_flag = 0; + if (!alternate_index_output && (flags & COMMIT_LOCK)) + csum_fsync_flag = CSUM_FSYNC; + + finalize_hashfile(f, istate->oid.hash, FSYNC_COMPONENT_INDEX, + CSUM_HASH_IN_STREAM | csum_fsync_flag); + if (close_tempfile_gently(tempfile)) { error(_("could not close '%s'"), get_tempfile_path(tempfile)); return -1; @@ -3115,7 +3122,7 @@ static int do_write_locked_index(struct index_state *istate, struct lock_file *l */ trace2_region_enter_printf("index", "do_write_index", the_repository, "%s", get_lock_file_path(lock)); - ret = do_write_index(istate, lock->tempfile, 0); + ret = do_write_index(istate, lock->tempfile, 0, flags); trace2_region_leave_printf("index", "do_write_index", the_repository, "%s", get_lock_file_path(lock)); @@ -3209,7 +3216,7 @@ static int clean_shared_index_files(const char *current_hex) } static int write_shared_index(struct index_state *istate, - struct tempfile **temp) + struct tempfile **temp, unsigned flags) { struct split_index *si = istate->split_index; int ret, was_full = !istate->sparse_index; @@ -3219,7 +3226,7 @@ static int write_shared_index(struct index_state *istate, trace2_region_enter_printf("index", "shared/do_write_index", the_repository, "%s", get_tempfile_path(*temp)); - ret = do_write_index(si->base, *temp, 1); + ret = do_write_index(si->base, *temp, 1, flags); trace2_region_leave_printf("index", "shared/do_write_index", the_repository, "%s", get_tempfile_path(*temp)); @@ -3328,7 +3335,7 @@ int write_locked_index(struct index_state *istate, struct lock_file *lock, ret = do_write_locked_index(istate, lock, flags); goto out; } - ret = write_shared_index(istate, &temp); + ret = write_shared_index(istate, &temp, flags); saved_errno = errno; if (is_tempfile_active(temp)) -- gitgitgadget ^ permalink raw reply related [flat|nested] 122+ messages in thread
* Re: [PATCH v2 0/3] A design for future-proofing fsync() configuration 2021-12-07 2:46 ` [PATCH v2 0/3] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget ` (2 preceding siblings ...) 2021-12-07 2:46 ` [PATCH v2 3/3] core.fsync: new option to harden the index Neeraj Singh via GitGitGadget @ 2021-12-07 11:56 ` Patrick Steinhardt 2021-12-08 0:44 ` Neeraj Singh 2021-12-09 0:57 ` [PATCH v3 0/4] " Neeraj K. Singh via GitGitGadget 4 siblings, 1 reply; 122+ messages in thread From: Patrick Steinhardt @ 2021-12-07 11:56 UTC (permalink / raw) To: Neeraj K. Singh via GitGitGadget Cc: git, rsbecker, bagasdotme, newren, avarab, nksingh85, Neeraj K. Singh [-- Attachment #1: Type: text/plain, Size: 27138 bytes --] On Tue, Dec 07, 2021 at 02:46:48AM +0000, Neeraj K. Singh via GitGitGadget wrote: > This is an implementation of an extensible configuration mechanism for > fsyncing persistent components of a repo. > > The main goals are to separate the "what" to sync from the "how". There are > now two settings: core.fsync - Control the 'what', including the index. > core.fsyncMethod - Control the 'how'. Currently we support writeout-only and > full fsync. > > Syncing of refs can be layered on top of core.fsync. And batch mode will be > layered on core.fsyncMethod. > > core.fsyncobjectfiles is removed and will issue a deprecation warning if > it's seen. > > I'd like to get agreement on this direction before submitting batch mode to > the list. The batch mode series is available to view at > https://github.com/neerajsi-msft/git/pull/1. > > Please see [1], [2], and [3] for discussions that led to this series. > > V2 changes: > > * Updated the documentation for core.fsyncmethod to be less certain. > writeout-only probably does not do the right thing on Linux. > * Split out the core.fsync=index change into its own commit. > * Rename REPO_COMPONENT to FSYNC_COMPONENT. This is really specific to > fsyncing, so the name should reflect that. > * Re-add missing Makefile change for SYNC_FILE_RANGE. > * Tested writeout-only mode, index syncing, and general config settings. > > [1] https://lore.kernel.org/git/211110.86r1bogg27.gmgdl@evledraar.gmail.com/ > [2] > https://lore.kernel.org/git/dd65718814011eb93ccc4428f9882e0f025224a6.1636029491.git.ps@pks.im/ > [3] > https://lore.kernel.org/git/pull.1076.git.git.1629856292.gitgitgadget@gmail.com/ While I bail from the question of whether we want to grant as much configurability to the user as this patch series does, I quite like the implementation. It feels rather straight-forward and it's easy to see how to extend it to support syncing of other subsystems like the loose refs. Thanks! Patrick > Neeraj Singh (3): > core.fsyncmethod: add writeout-only mode > core.fsync: introduce granular fsync control > core.fsync: new option to harden the index > > Documentation/config/core.txt | 35 +++++++++--- > Makefile | 6 ++ > builtin/fast-import.c | 2 +- > builtin/index-pack.c | 4 +- > builtin/pack-objects.c | 8 ++- > bulk-checkin.c | 5 +- > cache.h | 48 +++++++++++++++- > commit-graph.c | 3 +- > compat/mingw.h | 3 + > compat/win32/flush.c | 28 +++++++++ > config.c | 89 ++++++++++++++++++++++++++++- > config.mak.uname | 3 + > configure.ac | 8 +++ > contrib/buildsystems/CMakeLists.txt | 3 +- > csum-file.c | 5 +- > csum-file.h | 3 +- > environment.c | 3 +- > git-compat-util.h | 24 ++++++++ > midx.c | 3 +- > object-file.c | 3 +- > pack-bitmap-write.c | 3 +- > pack-write.c | 13 +++-- > read-cache.c | 19 ++++-- > wrapper.c | 56 ++++++++++++++++++ > write-or-die.c | 10 ++-- > 25 files changed, 344 insertions(+), 43 deletions(-) > create mode 100644 compat/win32/flush.c > > > base-commit: abe6bb3905392d5eb6b01fa6e54d7e784e0522aa > Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1093%2Fneerajsi-msft%2Fns%2Fcore-fsync-v2 > Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1093/neerajsi-msft/ns/core-fsync-v2 > Pull-Request: https://github.com/gitgitgadget/git/pull/1093 > > Range-diff vs v1: > > 1: 527380ddc3f ! 1: e79522cbdd4 fsync: add writeout-only mode for fsyncing repo data > @@ Metadata > Author: Neeraj Singh <neerajsi@microsoft.com> > > ## Commit message ## > - fsync: add writeout-only mode for fsyncing repo data > + core.fsyncmethod: add writeout-only mode > + > + This commit introduces the `core.fsyncmethod` configuration > + knob, which can currently be set to `fsync` or `writeout-only`. > > The new writeout-only mode attempts to tell the operating system to > flush its in-memory page cache to the storage hardware without issuing a > @@ Documentation/config/core.txt: core.whitespace:: > + using fsync and related primitives. > ++ > +* `fsync` uses the fsync() system call or platform equivalents. > -+* `writeout-only` issues requests to send the writes to the storage > -+ hardware, but does not send any FLUSH CACHE request. If the operating system > -+ does not support the required interfaces, this falls back to fsync(). > ++* `writeout-only` issues pagecache writeback requests, but depending on the > ++ filesystem and storage hardware, data added to the repository may not be > ++ durable in the event of a system crash. This is the default mode on macOS. > + > core.fsyncObjectFiles:: > This boolean will enable 'fsync()' when writing object files. > + > > + ## Makefile ## > +@@ Makefile: all:: > + # > + # Define HAVE_CLOCK_MONOTONIC if your platform has CLOCK_MONOTONIC. > + # > ++# Define HAVE_SYNC_FILE_RANGE if your platform has sync_file_range. > ++# > + # Define NEEDS_LIBRT if your platform requires linking with librt (glibc version > + # before 2.17) for clock_gettime and CLOCK_MONOTONIC. > + # > +@@ Makefile: ifdef HAVE_CLOCK_MONOTONIC > + BASIC_CFLAGS += -DHAVE_CLOCK_MONOTONIC > + endif > + > ++ifdef HAVE_SYNC_FILE_RANGE > ++ BASIC_CFLAGS += -DHAVE_SYNC_FILE_RANGE > ++endif > ++ > + ifdef NEEDS_LIBRT > + EXTLIBS += -lrt > + endif > + > ## cache.h ## > @@ cache.h: extern int read_replace_refs; > extern char *git_replace_ref_base; > 2: 23311a10142 ! 2: ff80a94bf9a core.fsync: introduce granular fsync control > @@ Commit message > knob which can be used to control how components of the > repository are made durable on disk. > > - This setting allows future extensibility of components > - that could be synced in two ways: > + This setting allows future extensibility of the list of > + syncable components: > * We issue a warning rather than an error for unrecognized > components, so new configs can be used with old Git versions. > * We support negation, so users can choose one of the default > aggregate options and then remove components that they don't > - want. > + want. The user would then harden any new components added in > + a Git version update. > > - This also support the common request of doing absolutely no > + This also supports the common request of doing absolutely no > fysncing with the `core.fsync=none` value, which is expected > to make the test suite faster. > > - This commit introduces the new ability for the user to harden > - the index, which is a requirement for being able to actually > - find a file that has been added to the repo and then deleted > - from the working tree. > - > Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> > > ## Documentation/config/core.txt ## > @@ Documentation/config/core.txt: core.whitespace:: > + hardened via the core.fsyncMethod when created or modified. You can > + disable hardening of any component by prefixing it with a '-'. Later > + items take precedence over earlier ones in the list. For example, > -+ `core.fsync=all,-index` means "harden everything except the index". > -+ Items that are not hardened may be lost in the event of an unclean > -+ system shutdown. > ++ `core.fsync=all,-pack-metadata` means "harden everything except pack > ++ metadata." Items that are not hardened may be lost in the event of an > ++ unclean system shutdown. > ++ > +* `none` disables fsync completely. This must be specified alone. > +* `loose-object` hardens objects added to the repo in loose-object form. > +* `pack` hardens objects added to the repo in packfile form. > +* `pack-metadata` hardens packfile bitmaps and indexes. > +* `commit-graph` hardens the commit graph file. > -+* `index` hardens the index when it is modified. > +* `objects` is an aggregate option that includes `loose-objects`, `pack`, > + `pack-metadata`, and `commit-graph`. > +* `default` is an aggregate option that is equivalent to `objects,-loose-object` > @@ Documentation/config/core.txt: core.whitespace:: > A value indicating the strategy Git will use to harden repository data > using fsync and related primitives. > @@ Documentation/config/core.txt: core.fsyncMethod:: > - hardware, but does not send any FLUSH CACHE request. If the operating system > - does not support the required interfaces, this falls back to fsync(). > + filesystem and storage hardware, data added to the repository may not be > + durable in the event of a system crash. This is the default mode on macOS. > > -core.fsyncObjectFiles:: > - This boolean will enable 'fsync()' when writing object files. > @@ builtin/fast-import.c: static void end_packfile(void) > > close_pack_windows(pack_data); > - finalize_hashfile(pack_file, cur_pack_oid.hash, 0); > -+ finalize_hashfile(pack_file, cur_pack_oid.hash, REPO_COMPONENT_PACK, 0); > ++ finalize_hashfile(pack_file, cur_pack_oid.hash, FSYNC_COMPONENT_PACK, 0); > fixup_pack_header_footer(pack_data->pack_fd, pack_data->hash, > pack_data->pack_name, object_count, > cur_pack_oid.hash, pack_size); > @@ builtin/index-pack.c: static void conclude_pack(int fix_thin_pack, const char *c > stop_progress_msg(&progress, msg.buf); > strbuf_release(&msg); > - finalize_hashfile(f, tail_hash, 0); > -+ finalize_hashfile(f, tail_hash, REPO_COMPONENT_PACK, 0); > ++ finalize_hashfile(f, tail_hash, FSYNC_COMPONENT_PACK, 0); > hashcpy(read_hash, pack_hash); > fixup_pack_header_footer(output_fd, pack_hash, > curr_pack, nr_objects, > @@ builtin/index-pack.c: static void final(const char *final_pack_name, const char > close(input_fd); > } else { > - fsync_or_die(output_fd, curr_pack_name); > -+ fsync_component_or_die(REPO_COMPONENT_PACK, output_fd, curr_pack_name); > ++ fsync_component_or_die(FSYNC_COMPONENT_PACK, output_fd, curr_pack_name); > err = close(output_fd); > if (err) > die_errno(_("error while closing pack file")); > @@ builtin/pack-objects.c: static void write_pack_file(void) > */ > if (pack_to_stdout) { > - finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_CLOSE); > -+ finalize_hashfile(f, hash, REPO_COMPONENT_NONE, > ++ finalize_hashfile(f, hash, FSYNC_COMPONENT_NONE, > + CSUM_HASH_IN_STREAM | CSUM_CLOSE); > } else if (nr_written == nr_remaining) { > - finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); > -+ finalize_hashfile(f, hash, REPO_COMPONENT_PACK, > ++ finalize_hashfile(f, hash, FSYNC_COMPONENT_PACK, > + CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); > } else { > - int fd = finalize_hashfile(f, hash, 0); > -+ int fd = finalize_hashfile(f, hash, REPO_COMPONENT_PACK, 0); > ++ int fd = finalize_hashfile(f, hash, FSYNC_COMPONENT_PACK, 0); > fixup_pack_header_footer(fd, hash, pack_tmp_name, > nr_written, hash, offset); > close(fd); > @@ bulk-checkin.c: static void finish_bulk_checkin(struct bulk_checkin_state *state > goto clear_exit; > } else if (state->nr_written == 1) { > - finalize_hashfile(state->f, hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); > -+ finalize_hashfile(state->f, hash, REPO_COMPONENT_PACK, > ++ finalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK, > + CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); > } else { > - int fd = finalize_hashfile(state->f, hash, 0); > -+ int fd = finalize_hashfile(state->f, hash, REPO_COMPONENT_PACK, 0); > ++ int fd = finalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK, 0); > fixup_pack_header_footer(fd, hash, state->pack_tmp_name, > state->nr_written, hash, > state->offset); > @@ cache.h: void reset_shared_repository(void); > -extern int fsync_object_files; > +/* > + * These values are used to help identify parts of a repository to fsync. > -+ * REPO_COMPONENT_NONE identifies data that will not be a persistent part of the > ++ * FSYNC_COMPONENT_NONE identifies data that will not be a persistent part of the > + * repository and so shouldn't be fsynced. > + */ > -+enum repo_component { > -+ REPO_COMPONENT_NONE = 0, > -+ REPO_COMPONENT_LOOSE_OBJECT = 1 << 0, > -+ REPO_COMPONENT_PACK = 1 << 1, > -+ REPO_COMPONENT_PACK_METADATA = 1 << 2, > -+ REPO_COMPONENT_COMMIT_GRAPH = 1 << 3, > -+ REPO_COMPONENT_INDEX = 1 << 4, > ++enum fsync_component { > ++ FSYNC_COMPONENT_NONE = 0, > ++ FSYNC_COMPONENT_LOOSE_OBJECT = 1 << 0, > ++ FSYNC_COMPONENT_PACK = 1 << 1, > ++ FSYNC_COMPONENT_PACK_METADATA = 1 << 2, > ++ FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3, > +}; > + > -+#define FSYNC_COMPONENTS_DEFAULT (REPO_COMPONENT_PACK | \ > -+ REPO_COMPONENT_PACK_METADATA | \ > -+ REPO_COMPONENT_COMMIT_GRAPH) > ++#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \ > ++ FSYNC_COMPONENT_PACK_METADATA | \ > ++ FSYNC_COMPONENT_COMMIT_GRAPH) > + > -+#define FSYNC_COMPONENTS_OBJECTS (REPO_COMPONENT_LOOSE_OBJECT | \ > -+ REPO_COMPONENT_PACK | \ > -+ REPO_COMPONENT_PACK_METADATA | \ > -+ REPO_COMPONENT_COMMIT_GRAPH) > ++#define FSYNC_COMPONENTS_OBJECTS (FSYNC_COMPONENT_LOOSE_OBJECT | \ > ++ FSYNC_COMPONENT_PACK | \ > ++ FSYNC_COMPONENT_PACK_METADATA | \ > ++ FSYNC_COMPONENT_COMMIT_GRAPH) > + > -+#define FSYNC_COMPONENTS_ALL (REPO_COMPONENT_LOOSE_OBJECT | \ > -+ REPO_COMPONENT_PACK | \ > -+ REPO_COMPONENT_PACK_METADATA | \ > -+ REPO_COMPONENT_COMMIT_GRAPH | \ > -+ REPO_COMPONENT_INDEX) > ++#define FSYNC_COMPONENTS_ALL (FSYNC_COMPONENT_LOOSE_OBJECT | \ > ++ FSYNC_COMPONENT_PACK | \ > ++ FSYNC_COMPONENT_PACK_METADATA | \ > ++ FSYNC_COMPONENT_COMMIT_GRAPH) > + > + > +/* > + * A bitmask indicating which components of the repo should be fsynced. > + */ > -+extern enum repo_component fsync_components; > ++extern enum fsync_component fsync_components; > > enum fsync_method { > FSYNC_METHOD_FSYNC, > @@ cache.h: int copy_file_with_time(const char *dst, const char *src, int mode); > void write_or_die(int fd, const void *buf, size_t count); > void fsync_or_die(int fd, const char *); > > -+inline void fsync_component_or_die(enum repo_component component, int fd, const char *msg) > ++inline void fsync_component_or_die(enum fsync_component component, int fd, const char *msg) > +{ > + if (fsync_components & component) > + fsync_or_die(fd, msg); > @@ commit-graph.c: static int write_commit_graph_file(struct write_commit_graph_con > > close_commit_graph(ctx->r->objects); > - finalize_hashfile(f, file_hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC); > -+ finalize_hashfile(f, file_hash, REPO_COMPONENT_COMMIT_GRAPH, > ++ finalize_hashfile(f, file_hash, FSYNC_COMPONENT_COMMIT_GRAPH, > + CSUM_HASH_IN_STREAM | CSUM_FSYNC); > free_chunkfile(cf); > > @@ config.c: static int git_parse_maybe_bool_text(const char *value) > > +static const struct fsync_component_entry { > + const char *name; > -+ enum repo_component component_bits; > ++ enum fsync_component component_bits; > +} fsync_component_table[] = { > -+ { "loose-object", REPO_COMPONENT_LOOSE_OBJECT }, > -+ { "pack", REPO_COMPONENT_PACK }, > -+ { "pack-metadata", REPO_COMPONENT_PACK_METADATA }, > -+ { "commit-graph", REPO_COMPONENT_COMMIT_GRAPH }, > -+ { "index", REPO_COMPONENT_INDEX }, > ++ { "loose-object", FSYNC_COMPONENT_LOOSE_OBJECT }, > ++ { "pack", FSYNC_COMPONENT_PACK }, > ++ { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA }, > ++ { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, > + { "objects", FSYNC_COMPONENTS_OBJECTS }, > + { "default", FSYNC_COMPONENTS_DEFAULT }, > + { "all", FSYNC_COMPONENTS_ALL }, > +}; > + > -+static enum repo_component parse_fsync_components(const char *var, const char *string) > ++static enum fsync_component parse_fsync_components(const char *var, const char *string) > +{ > -+ enum repo_component output = 0; > ++ enum fsync_component output = 0; > + > + if (!strcmp(string, "none")) > + return output; > @@ csum-file.c: static void free_hashfile(struct hashfile *f) > > -int finalize_hashfile(struct hashfile *f, unsigned char *result, unsigned int flags) > +int finalize_hashfile(struct hashfile *f, unsigned char *result, > -+ enum repo_component component, unsigned int flags) > ++ enum fsync_component component, unsigned int flags) > { > int fd; > > @@ csum-file.c: int finalize_hashfile(struct hashfile *f, unsigned char *result, un > die_errno("%s: sha1 file error on close", f->name); > > ## csum-file.h ## > +@@ > + #ifndef CSUM_FILE_H > + #define CSUM_FILE_H > + > ++#include "cache.h" > + #include "hash.h" > + > + struct progress; > @@ csum-file.h: int hashfile_truncate(struct hashfile *, struct hashfile_checkpoint *); > struct hashfile *hashfd(int fd, const char *name); > struct hashfile *hashfd_check(const char *name); > struct hashfile *hashfd_throughput(int fd, const char *name, struct progress *tp); > -int finalize_hashfile(struct hashfile *, unsigned char *, unsigned int); > -+int finalize_hashfile(struct hashfile *, unsigned char *, enum repo_component, unsigned int); > ++int finalize_hashfile(struct hashfile *, unsigned char *, enum fsync_component, unsigned int); > void hashwrite(struct hashfile *, const void *, unsigned int); > void hashflush(struct hashfile *f); > void crc32_begin(struct hashfile *); > @@ environment.c: const char *git_hooks_path; > int zlib_compression_level = Z_BEST_SPEED; > int pack_compression_level = Z_DEFAULT_COMPRESSION; > enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT; > -+enum repo_component fsync_components = FSYNC_COMPONENTS_DEFAULT; > ++enum fsync_component fsync_components = FSYNC_COMPONENTS_DEFAULT; > size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE; > size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT; > size_t delta_base_cache_limit = 96 * 1024 * 1024; > @@ midx.c: static int write_midx_internal(const char *object_dir, > write_chunkfile(cf, &ctx); > > - finalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM); > -+ finalize_hashfile(f, midx_hash, REPO_COMPONENT_PACK_METADATA, > ++ finalize_hashfile(f, midx_hash, FSYNC_COMPONENT_PACK_METADATA, > + CSUM_FSYNC | CSUM_HASH_IN_STREAM); > free_chunkfile(cf); > > @@ object-file.c: int hash_object_file(const struct git_hash_algo *algo, const void > { > - if (fsync_object_files) > - fsync_or_die(fd, "loose object file"); > -+ fsync_component_or_die(REPO_COMPONENT_LOOSE_OBJECT, fd, "loose object file"); > ++ fsync_component_or_die(FSYNC_COMPONENT_LOOSE_OBJECT, fd, "loose object file"); > if (close(fd) != 0) > die_errno(_("error when closing loose object file")); > } > @@ pack-bitmap-write.c: void bitmap_writer_finish(struct pack_idx_entry **index, > write_hash_cache(f, index, index_nr); > > - finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); > -+ finalize_hashfile(f, NULL, REPO_COMPONENT_PACK_METADATA, > ++ finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA, > + CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); > > if (adjust_shared_perm(tmp_file.buf)) > @@ pack-write.c: const char *write_idx_file(const char *index_name, struct pack_idx > hashwrite(f, sha1, the_hash_algo->rawsz); > - finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE | > - ((opts->flags & WRITE_IDX_VERIFY) > -+ finalize_hashfile(f, NULL, REPO_COMPONENT_PACK_METADATA, > +- ? 0 : CSUM_FSYNC)); > ++ finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA, > + CSUM_HASH_IN_STREAM | CSUM_CLOSE | > -+ ((opts->flags & WRITE_IDX_VERIFY) > - ? 0 : CSUM_FSYNC)); > ++ ((opts->flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC)); > return index_name; > } > + > @@ pack-write.c: const char *write_rev_file_order(const char *rev_name, > if (rev_name && adjust_shared_perm(rev_name) < 0) > die(_("failed to make %s readable"), rev_name); > > - finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE | > - ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC)); > -+ finalize_hashfile(f, NULL, REPO_COMPONENT_PACK_METADATA, > ++ finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA, > + CSUM_HASH_IN_STREAM | CSUM_CLOSE | > + ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC)); > > @@ pack-write.c: void fixup_pack_header_footer(int pack_fd, > the_hash_algo->final_fn(new_pack_hash, &new_hash_ctx); > write_or_die(pack_fd, new_pack_hash, the_hash_algo->rawsz); > - fsync_or_die(pack_fd, pack_name); > -+ fsync_component_or_die(REPO_COMPONENT_PACK, pack_fd, pack_name); > ++ fsync_component_or_die(FSYNC_COMPONENT_PACK, pack_fd, pack_name); > } > > char *index_pack_lockfile(int ip_out, int *is_well_formed) > > ## read-cache.c ## > -@@ read-cache.c: static int record_ieot(void) > - * rely on it. > - */ > - static int do_write_index(struct index_state *istate, struct tempfile *tempfile, > -- int strip_extensions) > -+ int strip_extensions, unsigned flags) > - { > - uint64_t start = getnanotime(); > - struct hashfile *f; > -@@ read-cache.c: static int do_write_index(struct index_state *istate, struct tempfile *tempfile, > - struct strbuf previous_name_buf = STRBUF_INIT, *previous_name; > - int drop_cache_tree = istate->drop_cache_tree; > - off_t offset; > -+ int csum_fsync_flag; > - int ieot_entries = 1; > - struct index_entry_offset_table *ieot = NULL; > - int nr, nr_threads; > @@ read-cache.c: static int do_write_index(struct index_state *istate, struct tempfile *tempfile, > return -1; > } > > - finalize_hashfile(f, istate->oid.hash, CSUM_HASH_IN_STREAM); > -+ csum_fsync_flag = 0; > -+ if (!alternate_index_output && (flags & COMMIT_LOCK)) > -+ csum_fsync_flag = CSUM_FSYNC; > -+ > -+ finalize_hashfile(f, istate->oid.hash, REPO_COMPONENT_INDEX, > -+ CSUM_HASH_IN_STREAM | csum_fsync_flag); > -+ > ++ finalize_hashfile(f, istate->oid.hash, FSYNC_COMPONENT_NONE, CSUM_HASH_IN_STREAM); > if (close_tempfile_gently(tempfile)) { > error(_("could not close '%s'"), get_tempfile_path(tempfile)); > return -1; > -@@ read-cache.c: static int do_write_locked_index(struct index_state *istate, struct lock_file *l > - */ > - trace2_region_enter_printf("index", "do_write_index", the_repository, > - "%s", get_lock_file_path(lock)); > -- ret = do_write_index(istate, lock->tempfile, 0); > -+ ret = do_write_index(istate, lock->tempfile, 0, flags); > - trace2_region_leave_printf("index", "do_write_index", the_repository, > - "%s", get_lock_file_path(lock)); > - > -@@ read-cache.c: static int clean_shared_index_files(const char *current_hex) > - } > - > - static int write_shared_index(struct index_state *istate, > -- struct tempfile **temp) > -+ struct tempfile **temp, unsigned flags) > - { > - struct split_index *si = istate->split_index; > - int ret, was_full = !istate->sparse_index; > -@@ read-cache.c: static int write_shared_index(struct index_state *istate, > - > - trace2_region_enter_printf("index", "shared/do_write_index", > - the_repository, "%s", get_tempfile_path(*temp)); > -- ret = do_write_index(si->base, *temp, 1); > -+ ret = do_write_index(si->base, *temp, 1, flags); > - trace2_region_leave_printf("index", "shared/do_write_index", > - the_repository, "%s", get_tempfile_path(*temp)); > - > -@@ read-cache.c: int write_locked_index(struct index_state *istate, struct lock_file *lock, > - ret = do_write_locked_index(istate, lock, flags); > - goto out; > - } > -- ret = write_shared_index(istate, &temp); > -+ ret = write_shared_index(istate, &temp, flags); > - > - saved_errno = errno; > - if (is_tempfile_active(temp)) > -: ----------- > 3: 86e39b8f8d1 core.fsync: new option to harden the index > > -- > gitgitgadget [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v2 0/3] A design for future-proofing fsync() configuration 2021-12-07 11:56 ` [PATCH v2 0/3] A design for future-proofing fsync() configuration Patrick Steinhardt @ 2021-12-08 0:44 ` Neeraj Singh 0 siblings, 0 replies; 122+ messages in thread From: Neeraj Singh @ 2021-12-08 0:44 UTC (permalink / raw) To: Patrick Steinhardt Cc: Neeraj K. Singh via GitGitGadget, Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Ævar Arnfjörð Bjarmason, Neeraj K. Singh On Tue, Dec 7, 2021 at 3:57 AM Patrick Steinhardt <ps@pks.im> wrote: > While I bail from the question of whether we want to grant as much > configurability to the user as this patch series does, I quite like the > implementation. It feels rather straight-forward and it's easy to see > how to extend it to support syncing of other subsystems like the loose > refs. > > Thanks! > > Patrick Thanks for the positive comment. I'm assuming that a major Git services like GitHub or GitLab would be able to take advantage of the granular options and knowledge of their hosting environment to choose the right values for any server-side git deployments. I'd probably turn off syncing for derived stuff like the commit-graph file and pack metadata. My underlying interest in all of these changes is to make Windows stop looking so bad (we're defaulting to core.fsyncobjectfiles=true). Batch mode should give similar safety and much more optimizable performance in our environment. Thanks, Neeraj ^ permalink raw reply [flat|nested] 122+ messages in thread
* [PATCH v3 0/4] A design for future-proofing fsync() configuration 2021-12-07 2:46 ` [PATCH v2 0/3] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget ` (3 preceding siblings ...) 2021-12-07 11:56 ` [PATCH v2 0/3] A design for future-proofing fsync() configuration Patrick Steinhardt @ 2021-12-09 0:57 ` Neeraj K. Singh via GitGitGadget 2021-12-09 0:57 ` [PATCH v3 1/4] core.fsyncmethod: add writeout-only mode Neeraj Singh via GitGitGadget ` (5 more replies) 4 siblings, 6 replies; 122+ messages in thread From: Neeraj K. Singh via GitGitGadget @ 2021-12-09 0:57 UTC (permalink / raw) To: git; +Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps, Neeraj K. Singh This is an implementation of an extensible configuration mechanism for fsyncing persistent components of a repo. The main goals are to separate the "what" to sync from the "how". There are now two settings: core.fsync - Control the 'what', including the index. core.fsyncMethod - Control the 'how'. Currently we support writeout-only and full fsync. Syncing of refs can be layered on top of core.fsync. And batch mode will be layered on core.fsyncMethod. core.fsyncobjectfiles is removed and will issue a deprecation warning if it's seen. I'd like to get agreement on this direction before submitting batch mode to the list. The batch mode series is available to view at https://github.com/neerajsi-msft/git/pull/1. Please see [1], [2], and [3] for discussions that led to this series. One major concern I'd voice that is adverse to this change: when a new persistent file is added to the Git repo, the person adding that file will need to update this configuration code and the documentation. Maybe this is the right thing to always think about, but the FSYNC_COMPONENT lists will be a single place that will receive a number of updates over time. Note: There's a minor conflict with ns/tmp-objdir. In object-file.c:close_loose_object we need to resolve it like this: if (!the_repository->objects->odb->will_destroy) fsync_component_or_die(FSYNC_COMPONENT_LOOSE_OBJECT, fd, "loose object file"); V3 changes: * Remove relative path from git-compat-util.h include [4]. * Updated newly added warning texts to have more context for localization [4]. * Fixed tab spacing in enum fsync_action * Moved the fsync looping out to a helper and do it consistently. [4] * Changed commit description to use camelCase for config names. [5] * Add an optional fourth patch with derived-metadata so that the user can exclude a forward-compatible set of things that should be recomputable given existing data. V2 changes: * Updated the documentation for core.fsyncmethod to be less certain. writeout-only probably does not do the right thing on Linux. * Split out the core.fsync=index change into its own commit. * Rename REPO_COMPONENT to FSYNC_COMPONENT. This is really specific to fsyncing, so the name should reflect that. * Re-add missing Makefile change for SYNC_FILE_RANGE. * Tested writeout-only mode, index syncing, and general config settings. [1] https://lore.kernel.org/git/211110.86r1bogg27.gmgdl@evledraar.gmail.com/ [2] https://lore.kernel.org/git/dd65718814011eb93ccc4428f9882e0f025224a6.1636029491.git.ps@pks.im/ [3] https://lore.kernel.org/git/pull.1076.git.git.1629856292.gitgitgadget@gmail.com/ [4] https://lore.kernel.org/git/CANQDOdf8C4-haK9=Q_J4Cid8bQALnmGDm=SvatRbaVf+tkzqLw@mail.gmail.com/ [5] https://lore.kernel.org/git/211207.861r2opplg.gmgdl@evledraar.gmail.com/ Neeraj Singh (4): core.fsyncmethod: add writeout-only mode core.fsync: introduce granular fsync control core.fsync: new option to harden the index core.fsync: add a `derived-metadata` aggregate option Documentation/config/core.txt | 35 ++++++++--- Makefile | 6 ++ builtin/fast-import.c | 2 +- builtin/index-pack.c | 4 +- builtin/pack-objects.c | 8 ++- bulk-checkin.c | 5 +- cache.h | 49 +++++++++++++++- commit-graph.c | 3 +- compat/mingw.h | 3 + compat/win32/flush.c | 28 +++++++++ config.c | 90 ++++++++++++++++++++++++++++- config.mak.uname | 3 + configure.ac | 8 +++ contrib/buildsystems/CMakeLists.txt | 3 +- csum-file.c | 5 +- csum-file.h | 3 +- environment.c | 3 +- git-compat-util.h | 24 ++++++++ midx.c | 3 +- object-file.c | 3 +- pack-bitmap-write.c | 3 +- pack-write.c | 13 +++-- read-cache.c | 19 ++++-- wrapper.c | 64 ++++++++++++++++++++ write-or-die.c | 10 ++-- 25 files changed, 354 insertions(+), 43 deletions(-) create mode 100644 compat/win32/flush.c base-commit: abe6bb3905392d5eb6b01fa6e54d7e784e0522aa Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1093%2Fneerajsi-msft%2Fns%2Fcore-fsync-v3 Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1093/neerajsi-msft/ns/core-fsync-v3 Pull-Request: https://github.com/gitgitgadget/git/pull/1093 Range-diff vs v2: 1: e79522cbdd4 ! 1: 15edfe51509 core.fsyncmethod: add writeout-only mode @@ Metadata ## Commit message ## core.fsyncmethod: add writeout-only mode - This commit introduces the `core.fsyncmethod` configuration + This commit introduces the `core.fsyncMethod` configuration knob, which can currently be set to `fsync` or `writeout-only`. The new writeout-only mode attempts to tell the operating system to @@ Commit message directive to the storage controller. This change updates fsync to do fcntl(F_FULLFSYNC) to make fsync actually durable. We maintain parity with existing behavior on Apple platforms by setting the default value - of the new core.fsyncmethod option. + of the new core.fsyncMethod option. Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> @@ compat/mingw.h: int mingw_getpagesize(void); ## compat/win32/flush.c (new) ## @@ -+#include "../../git-compat-util.h" ++#include "git-compat-util.h" +#include <winternl.h> +#include "lazyload.h" + @@ config.c: static int git_default_core_config(const char *var, const char *value, + else if (!strcmp(value, "writeout-only")) + fsync_method = FSYNC_METHOD_WRITEOUT_ONLY; + else -+ warning(_("unknown %s value '%s'"), var, value); ++ warning(_("ignoring unknown core.fsyncMethod value '%s'"), value); + + } + @@ git-compat-util.h: __attribute__((format (printf, 1, 2))) NORETURN +#endif + +enum fsync_action { -+ FSYNC_WRITEOUT_ONLY, -+ FSYNC_HARDWARE_FLUSH ++ FSYNC_WRITEOUT_ONLY, ++ FSYNC_HARDWARE_FLUSH +}; + +/* @@ wrapper.c: int xmkstemp_mode(char *filename_template, int mode) return fd; } ++/* ++ * Some platforms return EINTR from fsync. Since fsync is invoked in some ++ * cases by a wrapper that dies on failure, do not expose EINTR to callers. ++ */ ++static int fsync_loop(int fd) ++{ ++ int err; ++ ++ do { ++ err = fsync(fd); ++ } while (err < 0 && errno == EINTR); ++ return err; ++} ++ +int git_fsync(int fd, enum fsync_action action) +{ + switch (action) { @@ wrapper.c: int xmkstemp_mode(char *filename_template, int mode) + * on macOS, fsync just causes filesystem cache writeback but does not + * flush hardware caches. + */ -+ return fsync(fd); ++ return fsync_loop(fd); +#endif + +#ifdef HAVE_SYNC_FILE_RANGE @@ wrapper.c: int xmkstemp_mode(char *filename_template, int mode) + * case, since callers asking for a hardware flush may die if + * this function returns an error. + */ -+ for (;;) { -+ int err; +#ifdef __APPLE__ -+ err = fcntl(fd, F_FULLFSYNC); ++ return fcntl(fd, F_FULLFSYNC); +#else -+ err = fsync(fd); ++ return fsync_loop(fd); +#endif -+ if (err >= 0 || errno != EINTR) -+ return err; -+ } -+ + default: + BUG("unexpected git_fsync(%d) call", action); + } 2: ff80a94bf9a ! 2: 080be1a6f64 core.fsync: introduce granular fsync control @@ config.c: static int git_parse_maybe_bool_text(const char *value) + + if (!found) { + char *component = xstrndup(string, len); -+ warning(_("unknown %s value '%s'"), var, component); ++ warning(_("ignoring unknown core.fsync component '%s'"), component); + free(component); + } + 3: 86e39b8f8d1 = 3: 2207950beba core.fsync: new option to harden the index -: ----------- > 4: a830d177d4c core.fsync: add a `derived-metadata` aggregate option -- gitgitgadget ^ permalink raw reply [flat|nested] 122+ messages in thread
* [PATCH v3 1/4] core.fsyncmethod: add writeout-only mode 2021-12-09 0:57 ` [PATCH v3 0/4] " Neeraj K. Singh via GitGitGadget @ 2021-12-09 0:57 ` Neeraj Singh via GitGitGadget 2021-12-09 0:57 ` [PATCH v3 2/4] core.fsync: introduce granular fsync control Neeraj Singh via GitGitGadget ` (4 subsequent siblings) 5 siblings, 0 replies; 122+ messages in thread From: Neeraj Singh via GitGitGadget @ 2021-12-09 0:57 UTC (permalink / raw) To: git Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps, Neeraj K. Singh, Neeraj Singh From: Neeraj Singh <neerajsi@microsoft.com> This commit introduces the `core.fsyncMethod` configuration knob, which can currently be set to `fsync` or `writeout-only`. The new writeout-only mode attempts to tell the operating system to flush its in-memory page cache to the storage hardware without issuing a CACHE_FLUSH command to the storage controller. Writeout-only fsync is significantly faster than a vanilla fsync on common hardware, since data is written to a disk-side cache rather than all the way to a durable medium. Later changes in this patch series will take advantage of this primitive to implement batching of hardware flushes. When git_fsync is called with FSYNC_WRITEOUT_ONLY, it may fail and the caller is expected to do an ordinary fsync as needed. On Apple platforms, the fsync system call does not issue a CACHE_FLUSH directive to the storage controller. This change updates fsync to do fcntl(F_FULLFSYNC) to make fsync actually durable. We maintain parity with existing behavior on Apple platforms by setting the default value of the new core.fsyncMethod option. Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> --- Documentation/config/core.txt | 9 ++++ Makefile | 6 +++ cache.h | 7 ++++ compat/mingw.h | 3 ++ compat/win32/flush.c | 28 +++++++++++++ config.c | 12 ++++++ config.mak.uname | 3 ++ configure.ac | 8 ++++ contrib/buildsystems/CMakeLists.txt | 3 +- environment.c | 2 +- git-compat-util.h | 24 +++++++++++ wrapper.c | 64 +++++++++++++++++++++++++++++ write-or-die.c | 10 +++-- 13 files changed, 173 insertions(+), 6 deletions(-) create mode 100644 compat/win32/flush.c diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt index c04f62a54a1..dbb134f7136 100644 --- a/Documentation/config/core.txt +++ b/Documentation/config/core.txt @@ -547,6 +547,15 @@ core.whitespace:: is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent` errors. The default tab width is 8. Allowed values are 1 to 63. +core.fsyncMethod:: + A value indicating the strategy Git will use to harden repository data + using fsync and related primitives. ++ +* `fsync` uses the fsync() system call or platform equivalents. +* `writeout-only` issues pagecache writeback requests, but depending on the + filesystem and storage hardware, data added to the repository may not be + durable in the event of a system crash. This is the default mode on macOS. + core.fsyncObjectFiles:: This boolean will enable 'fsync()' when writing object files. + diff --git a/Makefile b/Makefile index d56c0e4aadc..cba024615c9 100644 --- a/Makefile +++ b/Makefile @@ -403,6 +403,8 @@ all:: # # Define HAVE_CLOCK_MONOTONIC if your platform has CLOCK_MONOTONIC. # +# Define HAVE_SYNC_FILE_RANGE if your platform has sync_file_range. +# # Define NEEDS_LIBRT if your platform requires linking with librt (glibc version # before 2.17) for clock_gettime and CLOCK_MONOTONIC. # @@ -1881,6 +1883,10 @@ ifdef HAVE_CLOCK_MONOTONIC BASIC_CFLAGS += -DHAVE_CLOCK_MONOTONIC endif +ifdef HAVE_SYNC_FILE_RANGE + BASIC_CFLAGS += -DHAVE_SYNC_FILE_RANGE +endif + ifdef NEEDS_LIBRT EXTLIBS += -lrt endif diff --git a/cache.h b/cache.h index eba12487b99..9cd60d94952 100644 --- a/cache.h +++ b/cache.h @@ -986,6 +986,13 @@ extern int read_replace_refs; extern char *git_replace_ref_base; extern int fsync_object_files; + +enum fsync_method { + FSYNC_METHOD_FSYNC, + FSYNC_METHOD_WRITEOUT_ONLY +}; + +extern enum fsync_method fsync_method; extern int core_preload_index; extern int precomposed_unicode; extern int protect_hfs; diff --git a/compat/mingw.h b/compat/mingw.h index c9a52ad64a6..6074a3d3ced 100644 --- a/compat/mingw.h +++ b/compat/mingw.h @@ -329,6 +329,9 @@ int mingw_getpagesize(void); #define getpagesize mingw_getpagesize #endif +int win32_fsync_no_flush(int fd); +#define fsync_no_flush win32_fsync_no_flush + struct rlimit { unsigned int rlim_cur; }; diff --git a/compat/win32/flush.c b/compat/win32/flush.c new file mode 100644 index 00000000000..6ca82f1dae5 --- /dev/null +++ b/compat/win32/flush.c @@ -0,0 +1,28 @@ +#include "git-compat-util.h" +#include <winternl.h> +#include "lazyload.h" + +int win32_fsync_no_flush(int fd) +{ + IO_STATUS_BLOCK io_status; + +#define FLUSH_FLAGS_FILE_DATA_ONLY 1 + + DECLARE_PROC_ADDR(ntdll.dll, NTSTATUS, NtFlushBuffersFileEx, + HANDLE FileHandle, ULONG Flags, PVOID Parameters, ULONG ParameterSize, + PIO_STATUS_BLOCK IoStatusBlock); + + if (!INIT_PROC_ADDR(NtFlushBuffersFileEx)) { + errno = ENOSYS; + return -1; + } + + memset(&io_status, 0, sizeof(io_status)); + if (NtFlushBuffersFileEx((HANDLE)_get_osfhandle(fd), FLUSH_FLAGS_FILE_DATA_ONLY, + NULL, 0, &io_status)) { + errno = EINVAL; + return -1; + } + + return 0; +} diff --git a/config.c b/config.c index c5873f3a706..139df71ba17 100644 --- a/config.c +++ b/config.c @@ -1490,6 +1490,18 @@ static int git_default_core_config(const char *var, const char *value, void *cb) return 0; } + if (!strcmp(var, "core.fsyncmethod")) { + if (!value) + return config_error_nonbool(var); + if (!strcmp(value, "fsync")) + fsync_method = FSYNC_METHOD_FSYNC; + else if (!strcmp(value, "writeout-only")) + fsync_method = FSYNC_METHOD_WRITEOUT_ONLY; + else + warning(_("ignoring unknown core.fsyncMethod value '%s'"), value); + + } + if (!strcmp(var, "core.fsyncobjectfiles")) { fsync_object_files = git_config_bool(var, value); return 0; diff --git a/config.mak.uname b/config.mak.uname index d0701f9beb0..774a09622d2 100644 --- a/config.mak.uname +++ b/config.mak.uname @@ -57,6 +57,7 @@ ifeq ($(uname_S),Linux) HAVE_CLOCK_MONOTONIC = YesPlease # -lrt is needed for clock_gettime on glibc <= 2.16 NEEDS_LIBRT = YesPlease + HAVE_SYNC_FILE_RANGE = YesPlease HAVE_GETDELIM = YesPlease FREAD_READS_DIRECTORIES = UnfortunatelyYes BASIC_CFLAGS += -DHAVE_SYSINFO @@ -453,6 +454,7 @@ endif CFLAGS = BASIC_CFLAGS = -nologo -I. -Icompat/vcbuild/include -DWIN32 -D_CONSOLE -DHAVE_STRING_H -D_CRT_SECURE_NO_WARNINGS -D_CRT_NONSTDC_NO_DEPRECATE COMPAT_OBJS = compat/msvc.o compat/winansi.o \ + compat/win32/flush.o \ compat/win32/path-utils.o \ compat/win32/pthread.o compat/win32/syslog.o \ compat/win32/trace2_win32_process_info.o \ @@ -628,6 +630,7 @@ ifeq ($(uname_S),MINGW) COMPAT_CFLAGS += -DSTRIP_EXTENSION=\".exe\" COMPAT_OBJS += compat/mingw.o compat/winansi.o \ compat/win32/trace2_win32_process_info.o \ + compat/win32/flush.o \ compat/win32/path-utils.o \ compat/win32/pthread.o compat/win32/syslog.o \ compat/win32/dirent.o diff --git a/configure.ac b/configure.ac index 5ee25ec95c8..6bd6bef1c44 100644 --- a/configure.ac +++ b/configure.ac @@ -1082,6 +1082,14 @@ AC_COMPILE_IFELSE([CLOCK_MONOTONIC_SRC], [AC_MSG_RESULT([no]) HAVE_CLOCK_MONOTONIC=]) GIT_CONF_SUBST([HAVE_CLOCK_MONOTONIC]) + +# +# Define HAVE_SYNC_FILE_RANGE=YesPlease if sync_file_range is available. +GIT_CHECK_FUNC(sync_file_range, + [HAVE_SYNC_FILE_RANGE=YesPlease], + [HAVE_SYNC_FILE_RANGE]) +GIT_CONF_SUBST([HAVE_SYNC_FILE_RANGE]) + # # Define NO_SETITIMER if you don't have setitimer. GIT_CHECK_FUNC(setitimer, diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt index 86b46114464..6d7bc16d054 100644 --- a/contrib/buildsystems/CMakeLists.txt +++ b/contrib/buildsystems/CMakeLists.txt @@ -261,7 +261,8 @@ if(CMAKE_SYSTEM_NAME STREQUAL "Windows") NOGDI OBJECT_CREATION_MODE=1 __USE_MINGW_ANSI_STDIO=0 USE_NED_ALLOCATOR OVERRIDE_STRDUP MMAP_PREVENTS_DELETE USE_WIN32_MMAP UNICODE _UNICODE HAVE_WPGMPTR ENSURE_MSYSTEM_IS_SET) - list(APPEND compat_SOURCES compat/mingw.c compat/winansi.c compat/win32/path-utils.c + list(APPEND compat_SOURCES compat/mingw.c compat/winansi.c + compat/win32/flush.c compat/win32/path-utils.c compat/win32/pthread.c compat/win32mmap.c compat/win32/syslog.c compat/win32/trace2_win32_process_info.c compat/win32/dirent.c compat/nedmalloc/nedmalloc.c compat/strdup.c) diff --git a/environment.c b/environment.c index 9da7f3c1a19..f9140e842cf 100644 --- a/environment.c +++ b/environment.c @@ -41,7 +41,7 @@ const char *git_attributes_file; const char *git_hooks_path; int zlib_compression_level = Z_BEST_SPEED; int pack_compression_level = Z_DEFAULT_COMPRESSION; -int fsync_object_files; +enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT; size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE; size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT; size_t delta_base_cache_limit = 96 * 1024 * 1024; diff --git a/git-compat-util.h b/git-compat-util.h index c6bd2a84e55..50db85a8610 100644 --- a/git-compat-util.h +++ b/git-compat-util.h @@ -1239,6 +1239,30 @@ __attribute__((format (printf, 1, 2))) NORETURN void BUG(const char *fmt, ...); #endif +#ifdef __APPLE__ +#define FSYNC_METHOD_DEFAULT FSYNC_METHOD_WRITEOUT_ONLY +#else +#define FSYNC_METHOD_DEFAULT FSYNC_METHOD_FSYNC +#endif + +enum fsync_action { + FSYNC_WRITEOUT_ONLY, + FSYNC_HARDWARE_FLUSH +}; + +/* + * Issues an fsync against the specified file according to the specified mode. + * + * FSYNC_WRITEOUT_ONLY attempts to use interfaces available on some operating + * systems to flush the OS cache without issuing a flush command to the storage + * controller. If those interfaces are unavailable, the function fails with + * ENOSYS. + * + * FSYNC_HARDWARE_FLUSH does an OS writeout and hardware flush to ensure that + * changes are durable. It is not expected to fail. + */ +int git_fsync(int fd, enum fsync_action action); + /* * Preserves errno, prints a message, but gives no warning for ENOENT. * Returns 0 on success, which includes trying to unlink an object that does diff --git a/wrapper.c b/wrapper.c index 36e12119d76..572f28f14ff 100644 --- a/wrapper.c +++ b/wrapper.c @@ -546,6 +546,70 @@ int xmkstemp_mode(char *filename_template, int mode) return fd; } +/* + * Some platforms return EINTR from fsync. Since fsync is invoked in some + * cases by a wrapper that dies on failure, do not expose EINTR to callers. + */ +static int fsync_loop(int fd) +{ + int err; + + do { + err = fsync(fd); + } while (err < 0 && errno == EINTR); + return err; +} + +int git_fsync(int fd, enum fsync_action action) +{ + switch (action) { + case FSYNC_WRITEOUT_ONLY: + +#ifdef __APPLE__ + /* + * on macOS, fsync just causes filesystem cache writeback but does not + * flush hardware caches. + */ + return fsync_loop(fd); +#endif + +#ifdef HAVE_SYNC_FILE_RANGE + /* + * On linux 2.6.17 and above, sync_file_range is the way to issue + * a writeback without a hardware flush. An offset of 0 and size of 0 + * indicates writeout of the entire file and the wait flags ensure that all + * dirty data is written to the disk (potentially in a disk-side cache) + * before we continue. + */ + + return sync_file_range(fd, 0, 0, SYNC_FILE_RANGE_WAIT_BEFORE | + SYNC_FILE_RANGE_WRITE | + SYNC_FILE_RANGE_WAIT_AFTER); +#endif + +#ifdef fsync_no_flush + return fsync_no_flush(fd); +#endif + + errno = ENOSYS; + return -1; + + case FSYNC_HARDWARE_FLUSH: + /* + * On some platforms fsync may return EINTR. Try again in this + * case, since callers asking for a hardware flush may die if + * this function returns an error. + */ +#ifdef __APPLE__ + return fcntl(fd, F_FULLFSYNC); +#else + return fsync_loop(fd); +#endif + default: + BUG("unexpected git_fsync(%d) call", action); + } +} + static int warn_if_unremovable(const char *op, const char *file, int rc) { int err; diff --git a/write-or-die.c b/write-or-die.c index 0b1ec8190b6..0702acdd5e8 100644 --- a/write-or-die.c +++ b/write-or-die.c @@ -57,10 +57,12 @@ void fprintf_or_die(FILE *f, const char *fmt, ...) void fsync_or_die(int fd, const char *msg) { - while (fsync(fd) < 0) { - if (errno != EINTR) - die_errno("fsync error on '%s'", msg); - } + if (fsync_method == FSYNC_METHOD_WRITEOUT_ONLY && + git_fsync(fd, FSYNC_WRITEOUT_ONLY) >= 0) + return; + + if (git_fsync(fd, FSYNC_HARDWARE_FLUSH) < 0) + die_errno("fsync error on '%s'", msg); } void write_or_die(int fd, const void *buf, size_t count) -- gitgitgadget ^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v3 2/4] core.fsync: introduce granular fsync control 2021-12-09 0:57 ` [PATCH v3 0/4] " Neeraj K. Singh via GitGitGadget 2021-12-09 0:57 ` [PATCH v3 1/4] core.fsyncmethod: add writeout-only mode Neeraj Singh via GitGitGadget @ 2021-12-09 0:57 ` Neeraj Singh via GitGitGadget 2021-12-09 0:57 ` [PATCH v3 3/4] core.fsync: new option to harden the index Neeraj Singh via GitGitGadget ` (3 subsequent siblings) 5 siblings, 0 replies; 122+ messages in thread From: Neeraj Singh via GitGitGadget @ 2021-12-09 0:57 UTC (permalink / raw) To: git Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps, Neeraj K. Singh, Neeraj Singh From: Neeraj Singh <neerajsi@microsoft.com> This commit introduces the `core.fsync` configuration knob which can be used to control how components of the repository are made durable on disk. This setting allows future extensibility of the list of syncable components: * We issue a warning rather than an error for unrecognized components, so new configs can be used with old Git versions. * We support negation, so users can choose one of the default aggregate options and then remove components that they don't want. The user would then harden any new components added in a Git version update. This also supports the common request of doing absolutely no fysncing with the `core.fsync=none` value, which is expected to make the test suite faster. Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> --- Documentation/config/core.txt | 27 +++++++++---- builtin/fast-import.c | 2 +- builtin/index-pack.c | 4 +- builtin/pack-objects.c | 8 ++-- bulk-checkin.c | 5 ++- cache.h | 39 +++++++++++++++++- commit-graph.c | 3 +- config.c | 76 ++++++++++++++++++++++++++++++++++- csum-file.c | 5 ++- csum-file.h | 3 +- environment.c | 1 + midx.c | 3 +- object-file.c | 3 +- pack-bitmap-write.c | 3 +- pack-write.c | 13 +++--- read-cache.c | 2 +- 16 files changed, 164 insertions(+), 33 deletions(-) diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt index dbb134f7136..4f1747ec871 100644 --- a/Documentation/config/core.txt +++ b/Documentation/config/core.txt @@ -547,6 +547,25 @@ core.whitespace:: is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent` errors. The default tab width is 8. Allowed values are 1 to 63. +core.fsync:: + A comma-separated list of parts of the repository which should be + hardened via the core.fsyncMethod when created or modified. You can + disable hardening of any component by prefixing it with a '-'. Later + items take precedence over earlier ones in the list. For example, + `core.fsync=all,-pack-metadata` means "harden everything except pack + metadata." Items that are not hardened may be lost in the event of an + unclean system shutdown. ++ +* `none` disables fsync completely. This must be specified alone. +* `loose-object` hardens objects added to the repo in loose-object form. +* `pack` hardens objects added to the repo in packfile form. +* `pack-metadata` hardens packfile bitmaps and indexes. +* `commit-graph` hardens the commit graph file. +* `objects` is an aggregate option that includes `loose-objects`, `pack`, + `pack-metadata`, and `commit-graph`. +* `default` is an aggregate option that is equivalent to `objects,-loose-object` +* `all` is an aggregate option that syncs all individual components above. + core.fsyncMethod:: A value indicating the strategy Git will use to harden repository data using fsync and related primitives. @@ -556,14 +575,6 @@ core.fsyncMethod:: filesystem and storage hardware, data added to the repository may not be durable in the event of a system crash. This is the default mode on macOS. -core.fsyncObjectFiles:: - This boolean will enable 'fsync()' when writing object files. -+ -This is a total waste of time and effort on a filesystem that orders -data writes properly, but can be useful for filesystems that do not use -journalling (traditional UNIX filesystems) or that only journal metadata -and not file contents (OS X's HFS+, or Linux ext3 with "data=writeback"). - core.preloadIndex:: Enable parallel index preload for operations like 'git diff' + diff --git a/builtin/fast-import.c b/builtin/fast-import.c index 20406f67754..e27a4580f85 100644 --- a/builtin/fast-import.c +++ b/builtin/fast-import.c @@ -856,7 +856,7 @@ static void end_packfile(void) struct tag *t; close_pack_windows(pack_data); - finalize_hashfile(pack_file, cur_pack_oid.hash, 0); + finalize_hashfile(pack_file, cur_pack_oid.hash, FSYNC_COMPONENT_PACK, 0); fixup_pack_header_footer(pack_data->pack_fd, pack_data->hash, pack_data->pack_name, object_count, cur_pack_oid.hash, pack_size); diff --git a/builtin/index-pack.c b/builtin/index-pack.c index c23d01de7dc..c32534c13b4 100644 --- a/builtin/index-pack.c +++ b/builtin/index-pack.c @@ -1286,7 +1286,7 @@ static void conclude_pack(int fix_thin_pack, const char *curr_pack, unsigned cha nr_objects - nr_objects_initial); stop_progress_msg(&progress, msg.buf); strbuf_release(&msg); - finalize_hashfile(f, tail_hash, 0); + finalize_hashfile(f, tail_hash, FSYNC_COMPONENT_PACK, 0); hashcpy(read_hash, pack_hash); fixup_pack_header_footer(output_fd, pack_hash, curr_pack, nr_objects, @@ -1508,7 +1508,7 @@ static void final(const char *final_pack_name, const char *curr_pack_name, if (!from_stdin) { close(input_fd); } else { - fsync_or_die(output_fd, curr_pack_name); + fsync_component_or_die(FSYNC_COMPONENT_PACK, output_fd, curr_pack_name); err = close(output_fd); if (err) die_errno(_("error while closing pack file")); diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 857be7826f3..916c55d6ce9 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -1204,11 +1204,13 @@ static void write_pack_file(void) * If so, rewrite it like in fast-import */ if (pack_to_stdout) { - finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_CLOSE); + finalize_hashfile(f, hash, FSYNC_COMPONENT_NONE, + CSUM_HASH_IN_STREAM | CSUM_CLOSE); } else if (nr_written == nr_remaining) { - finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); + finalize_hashfile(f, hash, FSYNC_COMPONENT_PACK, + CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); } else { - int fd = finalize_hashfile(f, hash, 0); + int fd = finalize_hashfile(f, hash, FSYNC_COMPONENT_PACK, 0); fixup_pack_header_footer(fd, hash, pack_tmp_name, nr_written, hash, offset); close(fd); diff --git a/bulk-checkin.c b/bulk-checkin.c index 8785b2ac806..a2cf9dcbc8d 100644 --- a/bulk-checkin.c +++ b/bulk-checkin.c @@ -53,9 +53,10 @@ static void finish_bulk_checkin(struct bulk_checkin_state *state) unlink(state->pack_tmp_name); goto clear_exit; } else if (state->nr_written == 1) { - finalize_hashfile(state->f, hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); + finalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK, + CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); } else { - int fd = finalize_hashfile(state->f, hash, 0); + int fd = finalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK, 0); fixup_pack_header_footer(fd, hash, state->pack_tmp_name, state->nr_written, hash, state->offset); diff --git a/cache.h b/cache.h index 9cd60d94952..d83fbaf2619 100644 --- a/cache.h +++ b/cache.h @@ -985,7 +985,38 @@ void reset_shared_repository(void); extern int read_replace_refs; extern char *git_replace_ref_base; -extern int fsync_object_files; +/* + * These values are used to help identify parts of a repository to fsync. + * FSYNC_COMPONENT_NONE identifies data that will not be a persistent part of the + * repository and so shouldn't be fsynced. + */ +enum fsync_component { + FSYNC_COMPONENT_NONE = 0, + FSYNC_COMPONENT_LOOSE_OBJECT = 1 << 0, + FSYNC_COMPONENT_PACK = 1 << 1, + FSYNC_COMPONENT_PACK_METADATA = 1 << 2, + FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3, +}; + +#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \ + FSYNC_COMPONENT_PACK_METADATA | \ + FSYNC_COMPONENT_COMMIT_GRAPH) + +#define FSYNC_COMPONENTS_OBJECTS (FSYNC_COMPONENT_LOOSE_OBJECT | \ + FSYNC_COMPONENT_PACK | \ + FSYNC_COMPONENT_PACK_METADATA | \ + FSYNC_COMPONENT_COMMIT_GRAPH) + +#define FSYNC_COMPONENTS_ALL (FSYNC_COMPONENT_LOOSE_OBJECT | \ + FSYNC_COMPONENT_PACK | \ + FSYNC_COMPONENT_PACK_METADATA | \ + FSYNC_COMPONENT_COMMIT_GRAPH) + + +/* + * A bitmask indicating which components of the repo should be fsynced. + */ +extern enum fsync_component fsync_components; enum fsync_method { FSYNC_METHOD_FSYNC, @@ -1747,6 +1778,12 @@ int copy_file_with_time(const char *dst, const char *src, int mode); void write_or_die(int fd, const void *buf, size_t count); void fsync_or_die(int fd, const char *); +inline void fsync_component_or_die(enum fsync_component component, int fd, const char *msg) +{ + if (fsync_components & component) + fsync_or_die(fd, msg); +} + ssize_t read_in_full(int fd, void *buf, size_t count); ssize_t write_in_full(int fd, const void *buf, size_t count); ssize_t pread_in_full(int fd, void *buf, size_t count, off_t offset); diff --git a/commit-graph.c b/commit-graph.c index 2706683acfe..c8a5dea4541 100644 --- a/commit-graph.c +++ b/commit-graph.c @@ -1939,7 +1939,8 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx) } close_commit_graph(ctx->r->objects); - finalize_hashfile(f, file_hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC); + finalize_hashfile(f, file_hash, FSYNC_COMPONENT_COMMIT_GRAPH, + CSUM_HASH_IN_STREAM | CSUM_FSYNC); free_chunkfile(cf); if (ctx->split) { diff --git a/config.c b/config.c index 139df71ba17..5ab381388f9 100644 --- a/config.c +++ b/config.c @@ -1213,6 +1213,73 @@ static int git_parse_maybe_bool_text(const char *value) return -1; } +static const struct fsync_component_entry { + const char *name; + enum fsync_component component_bits; +} fsync_component_table[] = { + { "loose-object", FSYNC_COMPONENT_LOOSE_OBJECT }, + { "pack", FSYNC_COMPONENT_PACK }, + { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA }, + { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, + { "objects", FSYNC_COMPONENTS_OBJECTS }, + { "default", FSYNC_COMPONENTS_DEFAULT }, + { "all", FSYNC_COMPONENTS_ALL }, +}; + +static enum fsync_component parse_fsync_components(const char *var, const char *string) +{ + enum fsync_component output = 0; + + if (!strcmp(string, "none")) + return output; + + while (string) { + int i; + size_t len; + const char *ep; + int negated = 0; + int found = 0; + + string = string + strspn(string, ", \t\n\r"); + ep = strchrnul(string, ','); + len = ep - string; + + if (*string == '-') { + negated = 1; + string++; + len--; + if (!len) + warning(_("invalid value for variable %s"), var); + } + + if (!len) + break; + + for (i = 0; i < ARRAY_SIZE(fsync_component_table); ++i) { + const struct fsync_component_entry *entry = &fsync_component_table[i]; + + if (strncmp(entry->name, string, len)) + continue; + + found = 1; + if (negated) + output &= ~entry->component_bits; + else + output |= entry->component_bits; + } + + if (!found) { + char *component = xstrndup(string, len); + warning(_("ignoring unknown core.fsync component '%s'"), component); + free(component); + } + + string = ep; + } + + return output; +} + int git_parse_maybe_bool(const char *value) { int v = git_parse_maybe_bool_text(value); @@ -1490,6 +1557,13 @@ static int git_default_core_config(const char *var, const char *value, void *cb) return 0; } + if (!strcmp(var, "core.fsync")) { + if (!value) + return config_error_nonbool(var); + fsync_components = parse_fsync_components(var, value); + return 0; + } + if (!strcmp(var, "core.fsyncmethod")) { if (!value) return config_error_nonbool(var); @@ -1503,7 +1577,7 @@ static int git_default_core_config(const char *var, const char *value, void *cb) } if (!strcmp(var, "core.fsyncobjectfiles")) { - fsync_object_files = git_config_bool(var, value); + warning(_("core.fsyncobjectfiles is deprecated; use core.fsync instead")); return 0; } diff --git a/csum-file.c b/csum-file.c index 26e8a6df44e..59ef3398ca2 100644 --- a/csum-file.c +++ b/csum-file.c @@ -58,7 +58,8 @@ static void free_hashfile(struct hashfile *f) free(f); } -int finalize_hashfile(struct hashfile *f, unsigned char *result, unsigned int flags) +int finalize_hashfile(struct hashfile *f, unsigned char *result, + enum fsync_component component, unsigned int flags) { int fd; @@ -69,7 +70,7 @@ int finalize_hashfile(struct hashfile *f, unsigned char *result, unsigned int fl if (flags & CSUM_HASH_IN_STREAM) flush(f, f->buffer, the_hash_algo->rawsz); if (flags & CSUM_FSYNC) - fsync_or_die(f->fd, f->name); + fsync_component_or_die(component, f->fd, f->name); if (flags & CSUM_CLOSE) { if (close(f->fd)) die_errno("%s: sha1 file error on close", f->name); diff --git a/csum-file.h b/csum-file.h index 291215b34eb..0d29f528fbc 100644 --- a/csum-file.h +++ b/csum-file.h @@ -1,6 +1,7 @@ #ifndef CSUM_FILE_H #define CSUM_FILE_H +#include "cache.h" #include "hash.h" struct progress; @@ -38,7 +39,7 @@ int hashfile_truncate(struct hashfile *, struct hashfile_checkpoint *); struct hashfile *hashfd(int fd, const char *name); struct hashfile *hashfd_check(const char *name); struct hashfile *hashfd_throughput(int fd, const char *name, struct progress *tp); -int finalize_hashfile(struct hashfile *, unsigned char *, unsigned int); +int finalize_hashfile(struct hashfile *, unsigned char *, enum fsync_component, unsigned int); void hashwrite(struct hashfile *, const void *, unsigned int); void hashflush(struct hashfile *f); void crc32_begin(struct hashfile *); diff --git a/environment.c b/environment.c index f9140e842cf..09905adecf9 100644 --- a/environment.c +++ b/environment.c @@ -42,6 +42,7 @@ const char *git_hooks_path; int zlib_compression_level = Z_BEST_SPEED; int pack_compression_level = Z_DEFAULT_COMPRESSION; enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT; +enum fsync_component fsync_components = FSYNC_COMPONENTS_DEFAULT; size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE; size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT; size_t delta_base_cache_limit = 96 * 1024 * 1024; diff --git a/midx.c b/midx.c index 837b46b2af5..882f91f7d57 100644 --- a/midx.c +++ b/midx.c @@ -1406,7 +1406,8 @@ static int write_midx_internal(const char *object_dir, write_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs); write_chunkfile(cf, &ctx); - finalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM); + finalize_hashfile(f, midx_hash, FSYNC_COMPONENT_PACK_METADATA, + CSUM_FSYNC | CSUM_HASH_IN_STREAM); free_chunkfile(cf); if (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP)) diff --git a/object-file.c b/object-file.c index eb972cdccd2..9d9c4a39e85 100644 --- a/object-file.c +++ b/object-file.c @@ -1809,8 +1809,7 @@ int hash_object_file(const struct git_hash_algo *algo, const void *buf, /* Finalize a file on disk, and close it. */ static void close_loose_object(int fd) { - if (fsync_object_files) - fsync_or_die(fd, "loose object file"); + fsync_component_or_die(FSYNC_COMPONENT_LOOSE_OBJECT, fd, "loose object file"); if (close(fd) != 0) die_errno(_("error when closing loose object file")); } diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c index 9c55c1531e1..c16e43d1669 100644 --- a/pack-bitmap-write.c +++ b/pack-bitmap-write.c @@ -719,7 +719,8 @@ void bitmap_writer_finish(struct pack_idx_entry **index, if (options & BITMAP_OPT_HASH_CACHE) write_hash_cache(f, index, index_nr); - finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); + finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA, + CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); if (adjust_shared_perm(tmp_file.buf)) die_errno("unable to make temporary bitmap file readable"); diff --git a/pack-write.c b/pack-write.c index a5846f3a346..51812cb1299 100644 --- a/pack-write.c +++ b/pack-write.c @@ -159,9 +159,9 @@ const char *write_idx_file(const char *index_name, struct pack_idx_entry **objec } hashwrite(f, sha1, the_hash_algo->rawsz); - finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE | - ((opts->flags & WRITE_IDX_VERIFY) - ? 0 : CSUM_FSYNC)); + finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA, + CSUM_HASH_IN_STREAM | CSUM_CLOSE | + ((opts->flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC)); return index_name; } @@ -281,8 +281,9 @@ const char *write_rev_file_order(const char *rev_name, if (rev_name && adjust_shared_perm(rev_name) < 0) die(_("failed to make %s readable"), rev_name); - finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE | - ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC)); + finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA, + CSUM_HASH_IN_STREAM | CSUM_CLOSE | + ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC)); return rev_name; } @@ -390,7 +391,7 @@ void fixup_pack_header_footer(int pack_fd, the_hash_algo->final_fn(partial_pack_hash, &old_hash_ctx); the_hash_algo->final_fn(new_pack_hash, &new_hash_ctx); write_or_die(pack_fd, new_pack_hash, the_hash_algo->rawsz); - fsync_or_die(pack_fd, pack_name); + fsync_component_or_die(FSYNC_COMPONENT_PACK, pack_fd, pack_name); } char *index_pack_lockfile(int ip_out, int *is_well_formed) diff --git a/read-cache.c b/read-cache.c index f3986596623..f3539681f49 100644 --- a/read-cache.c +++ b/read-cache.c @@ -3060,7 +3060,7 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile, return -1; } - finalize_hashfile(f, istate->oid.hash, CSUM_HASH_IN_STREAM); + finalize_hashfile(f, istate->oid.hash, FSYNC_COMPONENT_NONE, CSUM_HASH_IN_STREAM); if (close_tempfile_gently(tempfile)) { error(_("could not close '%s'"), get_tempfile_path(tempfile)); return -1; -- gitgitgadget ^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v3 3/4] core.fsync: new option to harden the index 2021-12-09 0:57 ` [PATCH v3 0/4] " Neeraj K. Singh via GitGitGadget 2021-12-09 0:57 ` [PATCH v3 1/4] core.fsyncmethod: add writeout-only mode Neeraj Singh via GitGitGadget 2021-12-09 0:57 ` [PATCH v3 2/4] core.fsync: introduce granular fsync control Neeraj Singh via GitGitGadget @ 2021-12-09 0:57 ` Neeraj Singh via GitGitGadget 2021-12-09 0:57 ` [PATCH v3 4/4] core.fsync: add a `derived-metadata` aggregate option Neeraj Singh via GitGitGadget ` (2 subsequent siblings) 5 siblings, 0 replies; 122+ messages in thread From: Neeraj Singh via GitGitGadget @ 2021-12-09 0:57 UTC (permalink / raw) To: git Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps, Neeraj K. Singh, Neeraj Singh From: Neeraj Singh <neerajsi@microsoft.com> This commit introduces the new ability for the user to harden the index. In the event of a system crash, the index must be durable for the user to actually find a file that has been added to the repo and then deleted from the working tree. We use the presence of the COMMIT_LOCK flag and absence of the alternate_index_output as a proxy for determining whether we're updating the persistent index of the repo or some temporary index. We don't sync these temporary indexes. Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> --- Documentation/config/core.txt | 1 + cache.h | 4 +++- config.c | 1 + read-cache.c | 19 +++++++++++++------ 4 files changed, 18 insertions(+), 7 deletions(-) diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt index 4f1747ec871..8e5b7a795ab 100644 --- a/Documentation/config/core.txt +++ b/Documentation/config/core.txt @@ -561,6 +561,7 @@ core.fsync:: * `pack` hardens objects added to the repo in packfile form. * `pack-metadata` hardens packfile bitmaps and indexes. * `commit-graph` hardens the commit graph file. +* `index` hardens the index when it is modified. * `objects` is an aggregate option that includes `loose-objects`, `pack`, `pack-metadata`, and `commit-graph`. * `default` is an aggregate option that is equivalent to `objects,-loose-object` diff --git a/cache.h b/cache.h index d83fbaf2619..4dc26d7b2c9 100644 --- a/cache.h +++ b/cache.h @@ -996,6 +996,7 @@ enum fsync_component { FSYNC_COMPONENT_PACK = 1 << 1, FSYNC_COMPONENT_PACK_METADATA = 1 << 2, FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3, + FSYNC_COMPONENT_INDEX = 1 << 4, }; #define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \ @@ -1010,7 +1011,8 @@ enum fsync_component { #define FSYNC_COMPONENTS_ALL (FSYNC_COMPONENT_LOOSE_OBJECT | \ FSYNC_COMPONENT_PACK | \ FSYNC_COMPONENT_PACK_METADATA | \ - FSYNC_COMPONENT_COMMIT_GRAPH) + FSYNC_COMPONENT_COMMIT_GRAPH | \ + FSYNC_COMPONENT_INDEX) /* diff --git a/config.c b/config.c index 5ab381388f9..b3e7006c68e 100644 --- a/config.c +++ b/config.c @@ -1221,6 +1221,7 @@ static const struct fsync_component_entry { { "pack", FSYNC_COMPONENT_PACK }, { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA }, { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, + { "index", FSYNC_COMPONENT_INDEX }, { "objects", FSYNC_COMPONENTS_OBJECTS }, { "default", FSYNC_COMPONENTS_DEFAULT }, { "all", FSYNC_COMPONENTS_ALL }, diff --git a/read-cache.c b/read-cache.c index f3539681f49..783cb3ea5db 100644 --- a/read-cache.c +++ b/read-cache.c @@ -2816,7 +2816,7 @@ static int record_ieot(void) * rely on it. */ static int do_write_index(struct index_state *istate, struct tempfile *tempfile, - int strip_extensions) + int strip_extensions, unsigned flags) { uint64_t start = getnanotime(); struct hashfile *f; @@ -2830,6 +2830,7 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile, struct strbuf previous_name_buf = STRBUF_INIT, *previous_name; int drop_cache_tree = istate->drop_cache_tree; off_t offset; + int csum_fsync_flag; int ieot_entries = 1; struct index_entry_offset_table *ieot = NULL; int nr, nr_threads; @@ -3060,7 +3061,13 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile, return -1; } - finalize_hashfile(f, istate->oid.hash, FSYNC_COMPONENT_NONE, CSUM_HASH_IN_STREAM); + csum_fsync_flag = 0; + if (!alternate_index_output && (flags & COMMIT_LOCK)) + csum_fsync_flag = CSUM_FSYNC; + + finalize_hashfile(f, istate->oid.hash, FSYNC_COMPONENT_INDEX, + CSUM_HASH_IN_STREAM | csum_fsync_flag); + if (close_tempfile_gently(tempfile)) { error(_("could not close '%s'"), get_tempfile_path(tempfile)); return -1; @@ -3115,7 +3122,7 @@ static int do_write_locked_index(struct index_state *istate, struct lock_file *l */ trace2_region_enter_printf("index", "do_write_index", the_repository, "%s", get_lock_file_path(lock)); - ret = do_write_index(istate, lock->tempfile, 0); + ret = do_write_index(istate, lock->tempfile, 0, flags); trace2_region_leave_printf("index", "do_write_index", the_repository, "%s", get_lock_file_path(lock)); @@ -3209,7 +3216,7 @@ static int clean_shared_index_files(const char *current_hex) } static int write_shared_index(struct index_state *istate, - struct tempfile **temp) + struct tempfile **temp, unsigned flags) { struct split_index *si = istate->split_index; int ret, was_full = !istate->sparse_index; @@ -3219,7 +3226,7 @@ static int write_shared_index(struct index_state *istate, trace2_region_enter_printf("index", "shared/do_write_index", the_repository, "%s", get_tempfile_path(*temp)); - ret = do_write_index(si->base, *temp, 1); + ret = do_write_index(si->base, *temp, 1, flags); trace2_region_leave_printf("index", "shared/do_write_index", the_repository, "%s", get_tempfile_path(*temp)); @@ -3328,7 +3335,7 @@ int write_locked_index(struct index_state *istate, struct lock_file *lock, ret = do_write_locked_index(istate, lock, flags); goto out; } - ret = write_shared_index(istate, &temp); + ret = write_shared_index(istate, &temp, flags); saved_errno = errno; if (is_tempfile_active(temp)) -- gitgitgadget ^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v3 4/4] core.fsync: add a `derived-metadata` aggregate option 2021-12-09 0:57 ` [PATCH v3 0/4] " Neeraj K. Singh via GitGitGadget ` (2 preceding siblings ...) 2021-12-09 0:57 ` [PATCH v3 3/4] core.fsync: new option to harden the index Neeraj Singh via GitGitGadget @ 2021-12-09 0:57 ` Neeraj Singh via GitGitGadget 2022-01-08 1:13 ` [PATCH v3 0/4] A design for future-proofing fsync() configuration Neeraj Singh 2022-02-01 3:33 ` [PATCH v4 " Neeraj K. Singh via GitGitGadget 5 siblings, 0 replies; 122+ messages in thread From: Neeraj Singh via GitGitGadget @ 2021-12-09 0:57 UTC (permalink / raw) To: git Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps, Neeraj K. Singh, Neeraj Singh From: Neeraj Singh <neerajsi@microsoft.com> This commit adds an aggregate option that currently includes the commit-graph file and pack metadata (indexes and bitmaps). The user may want to exclude this set from durability since they can be recomputed from other data if they wind up corrupt or missing. This is split out from the other patches in the series since it is an optional nice-to-have that might be controversial. Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> --- Documentation/config/core.txt | 6 +++--- cache.h | 7 ++++--- config.c | 1 + 3 files changed, 8 insertions(+), 6 deletions(-) diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt index 8e5b7a795ab..21092f3a4d1 100644 --- a/Documentation/config/core.txt +++ b/Documentation/config/core.txt @@ -562,9 +562,9 @@ core.fsync:: * `pack-metadata` hardens packfile bitmaps and indexes. * `commit-graph` hardens the commit graph file. * `index` hardens the index when it is modified. -* `objects` is an aggregate option that includes `loose-objects`, `pack`, - `pack-metadata`, and `commit-graph`. -* `default` is an aggregate option that is equivalent to `objects,-loose-object` +* `objects` is an aggregate option that includes `loose-objects` and `pack`. +* `derived-metadata` is an aggregate option that includes `pack-metadata` and `commit-graph`. +* `default` is an aggregate option that is equivalent to `objects,derived-metadata,-loose-object` * `all` is an aggregate option that syncs all individual components above. core.fsyncMethod:: diff --git a/cache.h b/cache.h index 4dc26d7b2c9..cc1c084242e 100644 --- a/cache.h +++ b/cache.h @@ -1004,9 +1004,10 @@ enum fsync_component { FSYNC_COMPONENT_COMMIT_GRAPH) #define FSYNC_COMPONENTS_OBJECTS (FSYNC_COMPONENT_LOOSE_OBJECT | \ - FSYNC_COMPONENT_PACK | \ - FSYNC_COMPONENT_PACK_METADATA | \ - FSYNC_COMPONENT_COMMIT_GRAPH) + FSYNC_COMPONENT_PACK) + +#define FSYNC_COMPONENTS_DERIVED_METADATA (FSYNC_COMPONENT_PACK_METADATA | \ + FSYNC_COMPONENT_COMMIT_GRAPH) #define FSYNC_COMPONENTS_ALL (FSYNC_COMPONENT_LOOSE_OBJECT | \ FSYNC_COMPONENT_PACK | \ diff --git a/config.c b/config.c index b3e7006c68e..d9ef3ef0060 100644 --- a/config.c +++ b/config.c @@ -1223,6 +1223,7 @@ static const struct fsync_component_entry { { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, { "index", FSYNC_COMPONENT_INDEX }, { "objects", FSYNC_COMPONENTS_OBJECTS }, + { "derived-metadata", FSYNC_COMPONENTS_DERIVED_METADATA }, { "default", FSYNC_COMPONENTS_DEFAULT }, { "all", FSYNC_COMPONENTS_ALL }, }; -- gitgitgadget ^ permalink raw reply related [flat|nested] 122+ messages in thread
* Re: [PATCH v3 0/4] A design for future-proofing fsync() configuration 2021-12-09 0:57 ` [PATCH v3 0/4] " Neeraj K. Singh via GitGitGadget ` (3 preceding siblings ...) 2021-12-09 0:57 ` [PATCH v3 4/4] core.fsync: add a `derived-metadata` aggregate option Neeraj Singh via GitGitGadget @ 2022-01-08 1:13 ` Neeraj Singh 2022-01-09 0:55 ` rsbecker 2022-02-01 3:33 ` [PATCH v4 " Neeraj K. Singh via GitGitGadget 5 siblings, 1 reply; 122+ messages in thread From: Neeraj Singh @ 2022-01-08 1:13 UTC (permalink / raw) To: Neeraj K. Singh via GitGitGadget Cc: Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Ævar Arnfjörð Bjarmason, Patrick Steinhardt, Neeraj K. Singh Hello Everyone, I wanted to revive this thread in the new year. To summarize the current state of affairs: * The current fsync patch series implements two new configuration options: core.fsync = <comma-separate list> -- select which repo components will be fsynced core.fsyncMethod = fsync|writeout-only -- select what form of fsyncing will be done * This patch series now ignores core.fsyncObjectFiles with a deprecation warning pointing the user at core.fsync. * There is a follow-on series that will extend the core.fsyncMethod to also include a `batch` mode that speeds up bulk operations by avoiding repeated disk cache flushes. * I developed the current mechanism after Ævar pointed out that the original `core.fsyncObjectFiles=batch` change would cause older versions of Git to die() when exposed to a new configuration. There were also several fsync changes floating around, including Patrick Steinhardts `core.fsyncRefFiles` change [1] and Eric Wong's `core.fsync = false` change [2]. * The biggest sticking points are in [3]. The fundamental disagreement is about whether core.fsync should look like: A) core.fsync = objects,commit-graph [current patch implementation] or B) core.fsync = objects core.fsync = commit-graph [Ævar's multivalued proposal]. I prefer sticking with (A) for reasons spelled out in the thread. I'm happy to re-litigate this discussion though. * There's also a sticking point about whether we should fsync when invoking pack-objects against stdout. I think that mostly reflects a missing comment in the code rather than a real disagreement. * Now that ew/test-wo-fsync has been integrated, there's some redundancy between core.fsync=none and Eric's patch. Open questions: 1) What format should we use for the core.fsync configuration to select individual repo components to sync? 2) Are we okay with deprecating core.fsyncObjectFiles in a single release with a warning? 3) Is it reasonable to expect people adding new persistent files to add and document new values of the core.fsync settings? Thanks, Neeraj [1] https://lore.kernel.org/git/20211030103950.M489266@dcvr/ [2] https://lore.kernel.org/git/20211028002102.19384-1-e@80x24.org/ [3] https://lore.kernel.org/git/211207.86wnkgo9fv.gmgdl@evledraar.gmail.com/ ^ permalink raw reply [flat|nested] 122+ messages in thread
* RE: [PATCH v3 0/4] A design for future-proofing fsync() configuration 2022-01-08 1:13 ` [PATCH v3 0/4] A design for future-proofing fsync() configuration Neeraj Singh @ 2022-01-09 0:55 ` rsbecker 2022-01-10 19:00 ` Neeraj Singh 0 siblings, 1 reply; 122+ messages in thread From: rsbecker @ 2022-01-09 0:55 UTC (permalink / raw) To: 'Neeraj Singh', 'Neeraj K. Singh via GitGitGadget' Cc: 'Git List', 'Bagas Sanjaya', 'Elijah Newren', 'Ævar Arnfjörð Bjarmason', 'Patrick Steinhardt', 'Neeraj K. Singh' On January 7, 2022 8:14 PM, Neeraj Singh wrote: > Hello Everyone, > I wanted to revive this thread in the new year. > > To summarize the current state of affairs: > * The current fsync patch series implements two new configuration options: > core.fsync = <comma-separate list> -- select which repo components will > be fsynced > core.fsyncMethod = fsync|writeout-only -- select what form of fsyncing > will be done > > * This patch series now ignores core.fsyncObjectFiles with a deprecation > warning pointing the user at core.fsync. > > * There is a follow-on series that will extend the core.fsyncMethod to also > include a `batch` mode that speeds up bulk operations by avoiding repeated > disk cache flushes. > > * I developed the current mechanism after Ævar pointed out that the original > `core.fsyncObjectFiles=batch` change would cause older versions of Git to > die() when exposed to a new configuration. There were also several fsync > changes floating around, including Patrick Steinhardts `core.fsyncRefFiles` > change [1] and Eric Wong's `core.fsync = false` change [2]. > > * The biggest sticking points are in [3]. The fundamental disagreement is > about whether core.fsync should look like: > A) core.fsync = objects,commit-graph [current patch implementation] > or > B) core.fsync = objects > core.fsync = commit-graph [Ævar's multivalued proposal]. > I prefer sticking with (A) for reasons spelled out in the thread. I'm happy to re- > litigate this discussion though. > > * There's also a sticking point about whether we should fsync when invoking > pack-objects against stdout. I think that mostly reflects a missing comment in > the code rather than a real disagreement. > > * Now that ew/test-wo-fsync has been integrated, there's some redundancy > between core.fsync=none and Eric's patch. > > Open questions: > 1) What format should we use for the core.fsync configuration to select > individual repo components to sync? > 2) Are we okay with deprecating core.fsyncObjectFiles in a single release with > a warning? > 3) Is it reasonable to expect people adding new persistent files to add and > document new values of the core.fsync settings? > > Thanks, > Neeraj > > [1] https://lore.kernel.org/git/20211030103950.M489266@dcvr/ > [2] https://lore.kernel.org/git/20211028002102.19384-1-e@80x24.org/ > [3] > https://lore.kernel.org/git/211207.86wnkgo9fv.gmgdl@evledraar.gmail.com/ Neeraj, Please remember that fsync() is operating system and version specific. You cannot make any assumptions about what is supported and what is not. I have recently had issues with git built on a recent operating system not running on a version from 2020. The proposed patches do not work, as I recall, in a portable manner, so caution is required making this change. You can expect this not to work on some platforms and some versions. Please account for that. Requiring users who are not aware of OS details to configure git to function at all is a bad move, in my view - which has not changed since last time. Thanks, Randall ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v3 0/4] A design for future-proofing fsync() configuration 2022-01-09 0:55 ` rsbecker @ 2022-01-10 19:00 ` Neeraj Singh 0 siblings, 0 replies; 122+ messages in thread From: Neeraj Singh @ 2022-01-10 19:00 UTC (permalink / raw) To: Randall S. Becker Cc: Neeraj K. Singh via GitGitGadget, Git List, Bagas Sanjaya, Elijah Newren, Ævar Arnfjörð Bjarmason, Patrick Steinhardt, Neeraj K. Singh On Sat, Jan 8, 2022 at 4:55 PM <rsbecker@nexbridge.com> wrote: > > Please remember that fsync() is operating system and version specific. You cannot make any assumptions about what is supported and what is not. I have recently had issues with git built on a recent operating system not running on a version from 2020. The proposed patches do not work, as I recall, in a portable manner, so caution is required making this change. You can expect this not to work on some platforms and some versions. Please account for that. Requiring users who are not aware of OS details to configure git to function at all is a bad move, in my view - which has not changed since last time. > There was already an implied configuration of fsync in the Git codebase. None of the defaults are changing--assuming that a user does not explicitly configure the core.fsync setting, Git should work the same as it always has. I don't believe the current patch series introduces any new incompatibilities. Thanks, Neeraj ^ permalink raw reply [flat|nested] 122+ messages in thread
* [PATCH v4 0/4] A design for future-proofing fsync() configuration 2021-12-09 0:57 ` [PATCH v3 0/4] " Neeraj K. Singh via GitGitGadget ` (4 preceding siblings ...) 2022-01-08 1:13 ` [PATCH v3 0/4] A design for future-proofing fsync() configuration Neeraj Singh @ 2022-02-01 3:33 ` Neeraj K. Singh via GitGitGadget 2022-02-01 3:33 ` [PATCH v4 1/4] core.fsyncmethod: add writeout-only mode Neeraj Singh via GitGitGadget ` (4 more replies) 5 siblings, 5 replies; 122+ messages in thread From: Neeraj K. Singh via GitGitGadget @ 2022-02-01 3:33 UTC (permalink / raw) To: git; +Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps, Neeraj K. Singh This is an implementation of an extensible configuration mechanism for fsyncing persistent components of a repo. The main goals are to separate the "what" to sync from the "how". There are now two settings: core.fsync - Control the 'what', including the index. core.fsyncMethod - Control the 'how'. Currently we support writeout-only and full fsync. Syncing of refs can be layered on top of core.fsync. And batch mode will be layered on core.fsyncMethod. core.fsyncObjectfiles is removed and will issue a deprecation warning if it's seen. I'd like to get agreement on this direction before submitting batch mode to the list. The batch mode series is available to view at https://github.com/gitgitgadget/git/pull/1134 Please see [1], [2], and [3] for discussions that led to this series. After this change, new persistent data files added to the repo will need to be added to the fsync_component enum and documented in the Documentation/config/core.txt text. V4 changes: * Rebase onto master at b23dac905bd. * Add a comment to write_pack_file indicating why we don't fsync when writing to stdout. * I kept the configuration schema as-is rather than switching to multi-value. The thinking here is that a stateless last-one-wins config schema (comma separated) will make it easier to achieve some holistic self-consistent fsync configuration for a particular repo. V3 changes: * Remove relative path from git-compat-util.h include [4]. * Updated newly added warning texts to have more context for localization [4]. * Fixed tab spacing in enum fsync_action * Moved the fsync looping out to a helper and do it consistently. [4] * Changed commit description to use camelCase for config names. [5] * Add an optional fourth patch with derived-metadata so that the user can exclude a forward-compatible set of things that should be recomputable given existing data. V2 changes: * Updated the documentation for core.fsyncmethod to be less certain. writeout-only probably does not do the right thing on Linux. * Split out the core.fsync=index change into its own commit. * Rename REPO_COMPONENT to FSYNC_COMPONENT. This is really specific to fsyncing, so the name should reflect that. * Re-add missing Makefile change for SYNC_FILE_RANGE. * Tested writeout-only mode, index syncing, and general config settings. [1] https://lore.kernel.org/git/211110.86r1bogg27.gmgdl@evledraar.gmail.com/ [2] https://lore.kernel.org/git/dd65718814011eb93ccc4428f9882e0f025224a6.1636029491.git.ps@pks.im/ [3] https://lore.kernel.org/git/pull.1076.git.git.1629856292.gitgitgadget@gmail.com/ [4] https://lore.kernel.org/git/CANQDOdf8C4-haK9=Q_J4Cid8bQALnmGDm=SvatRbaVf+tkzqLw@mail.gmail.com/ [5] https://lore.kernel.org/git/211207.861r2opplg.gmgdl@evledraar.gmail.com/ Neeraj Singh (4): core.fsyncmethod: add writeout-only mode core.fsync: introduce granular fsync control core.fsync: new option to harden the index core.fsync: add a `derived-metadata` aggregate option Documentation/config/core.txt | 35 ++++++++--- Makefile | 6 ++ builtin/fast-import.c | 2 +- builtin/index-pack.c | 4 +- builtin/pack-objects.c | 24 +++++--- bulk-checkin.c | 5 +- cache.h | 49 +++++++++++++++- commit-graph.c | 3 +- compat/mingw.h | 3 + compat/win32/flush.c | 28 +++++++++ config.c | 90 ++++++++++++++++++++++++++++- config.mak.uname | 3 + configure.ac | 8 +++ contrib/buildsystems/CMakeLists.txt | 3 +- csum-file.c | 5 +- csum-file.h | 3 +- environment.c | 3 +- git-compat-util.h | 24 ++++++++ midx.c | 3 +- object-file.c | 3 +- pack-bitmap-write.c | 3 +- pack-write.c | 13 +++-- read-cache.c | 19 ++++-- wrapper.c | 64 ++++++++++++++++++++ write-or-die.c | 11 ++-- 25 files changed, 367 insertions(+), 47 deletions(-) create mode 100644 compat/win32/flush.c base-commit: b23dac905bde28da47543484320db16312c87551 Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1093%2Fneerajsi-msft%2Fns%2Fcore-fsync-v4 Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1093/neerajsi-msft/ns/core-fsync-v4 Pull-Request: https://github.com/gitgitgadget/git/pull/1093 Range-diff vs v3: 1: 15edfe51509 ! 1: 51a218d100d core.fsyncmethod: add writeout-only mode @@ Makefile: ifdef HAVE_CLOCK_MONOTONIC endif ## cache.h ## -@@ cache.h: extern int read_replace_refs; - extern char *git_replace_ref_base; +@@ cache.h: extern char *git_replace_ref_base; extern int fsync_object_files; + extern int use_fsync; + +enum fsync_method { + FSYNC_METHOD_FSYNC, @@ compat/win32/flush.c (new) + +#define FLUSH_FLAGS_FILE_DATA_ONLY 1 + -+ DECLARE_PROC_ADDR(ntdll.dll, NTSTATUS, NtFlushBuffersFileEx, ++ DECLARE_PROC_ADDR(ntdll.dll, NTSTATUS, NTAPI, NtFlushBuffersFileEx, + HANDLE FileHandle, ULONG Flags, PVOID Parameters, ULONG ParameterSize, + PIO_STATUS_BLOCK IoStatusBlock); + @@ contrib/buildsystems/CMakeLists.txt: if(CMAKE_SYSTEM_NAME STREQUAL "Windows") compat/nedmalloc/nedmalloc.c compat/strdup.c) ## environment.c ## -@@ environment.c: const char *git_attributes_file; - const char *git_hooks_path; - int zlib_compression_level = Z_BEST_SPEED; +@@ environment.c: int zlib_compression_level = Z_BEST_SPEED; int pack_compression_level = Z_DEFAULT_COMPRESSION; --int fsync_object_files; + int fsync_object_files; + int use_fsync = -1; +enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT; size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE; size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT; @@ wrapper.c: int xmkstemp_mode(char *filename_template, int mode) int err; ## write-or-die.c ## -@@ write-or-die.c: void fprintf_or_die(FILE *f, const char *fmt, ...) - - void fsync_or_die(int fd, const char *msg) - { +@@ write-or-die.c: void fsync_or_die(int fd, const char *msg) + use_fsync = git_env_bool("GIT_TEST_FSYNC", 1); + if (!use_fsync) + return; - while (fsync(fd) < 0) { - if (errno != EINTR) - die_errno("fsync error on '%s'", msg); - } ++ + if (fsync_method == FSYNC_METHOD_WRITEOUT_ONLY && + git_fsync(fd, FSYNC_WRITEOUT_ONLY) >= 0) + return; 2: 080be1a6f64 ! 2: 7a164ba9571 core.fsync: introduce granular fsync control @@ builtin/index-pack.c: static void final(const char *final_pack_name, const char ## builtin/pack-objects.c ## @@ builtin/pack-objects.c: static void write_pack_file(void) - * If so, rewrite it like in fast-import - */ + display_progress(progress_state, written); + } + +- /* +- * Did we write the wrong # entries in the header? +- * If so, rewrite it like in fast-import +- */ if (pack_to_stdout) { - finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_CLOSE); ++ /* ++ * We never fsync when writing to stdout since we may ++ * not be writing to an actual pack file. For instance, ++ * the upload-pack code passes a pipe here. Calling ++ * fsync on a pipe results in unnecessary ++ * synchronization with the reader on some platforms. ++ */ + finalize_hashfile(f, hash, FSYNC_COMPONENT_NONE, + CSUM_HASH_IN_STREAM | CSUM_CLOSE); } else if (nr_written == nr_remaining) { @@ builtin/pack-objects.c: static void write_pack_file(void) + CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); } else { - int fd = finalize_hashfile(f, hash, 0); ++ /* ++ * If we wrote the wrong number of entries in the ++ * header, rewrite it like in fast-import. ++ */ ++ + int fd = finalize_hashfile(f, hash, FSYNC_COMPONENT_PACK, 0); fixup_pack_header_footer(fd, hash, pack_tmp_name, nr_written, hash, offset); @@ cache.h: void reset_shared_repository(void); extern char *git_replace_ref_base; -extern int fsync_object_files; +-extern int use_fsync; +/* + * These values are used to help identify parts of a repository to fsync. + * FSYNC_COMPONENT_NONE identifies data that will not be a persistent part of the + * repository and so shouldn't be fsynced. + */ +enum fsync_component { -+ FSYNC_COMPONENT_NONE = 0, ++ FSYNC_COMPONENT_NONE, + FSYNC_COMPONENT_LOOSE_OBJECT = 1 << 0, + FSYNC_COMPONENT_PACK = 1 << 1, + FSYNC_COMPONENT_PACK_METADATA = 1 << 2, @@ cache.h: void reset_shared_repository(void); enum fsync_method { FSYNC_METHOD_FSYNC, +@@ cache.h: enum fsync_method { + }; + + extern enum fsync_method fsync_method; ++extern int use_fsync; + extern int core_preload_index; + extern int precomposed_unicode; + extern int protect_hfs; @@ cache.h: int copy_file_with_time(const char *dst, const char *src, int mode); void write_or_die(int fd, const void *buf, size_t count); void fsync_or_die(int fd, const char *); @@ csum-file.h: int hashfile_truncate(struct hashfile *, struct hashfile_checkpoint void crc32_begin(struct hashfile *); ## environment.c ## -@@ environment.c: const char *git_hooks_path; +@@ environment.c: const char *git_attributes_file; + const char *git_hooks_path; int zlib_compression_level = Z_BEST_SPEED; int pack_compression_level = Z_DEFAULT_COMPRESSION; +-int fsync_object_files; + int use_fsync = -1; enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT; +enum fsync_component fsync_components = FSYNC_COMPONENTS_DEFAULT; size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE; @@ midx.c: static int write_midx_internal(const char *object_dir, ## object-file.c ## @@ object-file.c: int hash_object_file(const struct git_hash_algo *algo, const void *buf, - /* Finalize a file on disk, and close it. */ static void close_loose_object(int fd) { -- if (fsync_object_files) -- fsync_or_die(fd, "loose object file"); -+ fsync_component_or_die(FSYNC_COMPONENT_LOOSE_OBJECT, fd, "loose object file"); + if (!the_repository->objects->odb->will_destroy) { +- if (fsync_object_files) +- fsync_or_die(fd, "loose object file"); ++ fsync_component_or_die(FSYNC_COMPONENT_LOOSE_OBJECT, fd, "loose object file"); + } + if (close(fd) != 0) - die_errno(_("error when closing loose object file")); - } ## pack-bitmap-write.c ## @@ pack-bitmap-write.c: void bitmap_writer_finish(struct pack_idx_entry **index, 3: 2207950beba = 3: f217dba77a1 core.fsync: new option to harden the index 4: a830d177d4c = 4: 5c22a41c1f3 core.fsync: add a `derived-metadata` aggregate option -- gitgitgadget ^ permalink raw reply [flat|nested] 122+ messages in thread
* [PATCH v4 1/4] core.fsyncmethod: add writeout-only mode 2022-02-01 3:33 ` [PATCH v4 " Neeraj K. Singh via GitGitGadget @ 2022-02-01 3:33 ` Neeraj Singh via GitGitGadget 2022-02-01 3:33 ` [PATCH v4 2/4] core.fsync: introduce granular fsync control Neeraj Singh via GitGitGadget ` (3 subsequent siblings) 4 siblings, 0 replies; 122+ messages in thread From: Neeraj Singh via GitGitGadget @ 2022-02-01 3:33 UTC (permalink / raw) To: git Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps, Neeraj K. Singh, Neeraj Singh From: Neeraj Singh <neerajsi@microsoft.com> This commit introduces the `core.fsyncMethod` configuration knob, which can currently be set to `fsync` or `writeout-only`. The new writeout-only mode attempts to tell the operating system to flush its in-memory page cache to the storage hardware without issuing a CACHE_FLUSH command to the storage controller. Writeout-only fsync is significantly faster than a vanilla fsync on common hardware, since data is written to a disk-side cache rather than all the way to a durable medium. Later changes in this patch series will take advantage of this primitive to implement batching of hardware flushes. When git_fsync is called with FSYNC_WRITEOUT_ONLY, it may fail and the caller is expected to do an ordinary fsync as needed. On Apple platforms, the fsync system call does not issue a CACHE_FLUSH directive to the storage controller. This change updates fsync to do fcntl(F_FULLFSYNC) to make fsync actually durable. We maintain parity with existing behavior on Apple platforms by setting the default value of the new core.fsyncMethod option. Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> --- Documentation/config/core.txt | 9 ++++ Makefile | 6 +++ cache.h | 7 ++++ compat/mingw.h | 3 ++ compat/win32/flush.c | 28 +++++++++++++ config.c | 12 ++++++ config.mak.uname | 3 ++ configure.ac | 8 ++++ contrib/buildsystems/CMakeLists.txt | 3 +- environment.c | 1 + git-compat-util.h | 24 +++++++++++ wrapper.c | 64 +++++++++++++++++++++++++++++ write-or-die.c | 11 +++-- 13 files changed, 174 insertions(+), 5 deletions(-) create mode 100644 compat/win32/flush.c diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt index c04f62a54a1..dbb134f7136 100644 --- a/Documentation/config/core.txt +++ b/Documentation/config/core.txt @@ -547,6 +547,15 @@ core.whitespace:: is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent` errors. The default tab width is 8. Allowed values are 1 to 63. +core.fsyncMethod:: + A value indicating the strategy Git will use to harden repository data + using fsync and related primitives. ++ +* `fsync` uses the fsync() system call or platform equivalents. +* `writeout-only` issues pagecache writeback requests, but depending on the + filesystem and storage hardware, data added to the repository may not be + durable in the event of a system crash. This is the default mode on macOS. + core.fsyncObjectFiles:: This boolean will enable 'fsync()' when writing object files. + diff --git a/Makefile b/Makefile index 5580859afdb..1eff9953280 100644 --- a/Makefile +++ b/Makefile @@ -405,6 +405,8 @@ all:: # # Define HAVE_CLOCK_MONOTONIC if your platform has CLOCK_MONOTONIC. # +# Define HAVE_SYNC_FILE_RANGE if your platform has sync_file_range. +# # Define NEEDS_LIBRT if your platform requires linking with librt (glibc version # before 2.17) for clock_gettime and CLOCK_MONOTONIC. # @@ -1892,6 +1894,10 @@ ifdef HAVE_CLOCK_MONOTONIC BASIC_CFLAGS += -DHAVE_CLOCK_MONOTONIC endif +ifdef HAVE_SYNC_FILE_RANGE + BASIC_CFLAGS += -DHAVE_SYNC_FILE_RANGE +endif + ifdef NEEDS_LIBRT EXTLIBS += -lrt endif diff --git a/cache.h b/cache.h index 281f00ab1b1..37a32034b2f 100644 --- a/cache.h +++ b/cache.h @@ -995,6 +995,13 @@ extern char *git_replace_ref_base; extern int fsync_object_files; extern int use_fsync; + +enum fsync_method { + FSYNC_METHOD_FSYNC, + FSYNC_METHOD_WRITEOUT_ONLY +}; + +extern enum fsync_method fsync_method; extern int core_preload_index; extern int precomposed_unicode; extern int protect_hfs; diff --git a/compat/mingw.h b/compat/mingw.h index c9a52ad64a6..6074a3d3ced 100644 --- a/compat/mingw.h +++ b/compat/mingw.h @@ -329,6 +329,9 @@ int mingw_getpagesize(void); #define getpagesize mingw_getpagesize #endif +int win32_fsync_no_flush(int fd); +#define fsync_no_flush win32_fsync_no_flush + struct rlimit { unsigned int rlim_cur; }; diff --git a/compat/win32/flush.c b/compat/win32/flush.c new file mode 100644 index 00000000000..291f90ea940 --- /dev/null +++ b/compat/win32/flush.c @@ -0,0 +1,28 @@ +#include "git-compat-util.h" +#include <winternl.h> +#include "lazyload.h" + +int win32_fsync_no_flush(int fd) +{ + IO_STATUS_BLOCK io_status; + +#define FLUSH_FLAGS_FILE_DATA_ONLY 1 + + DECLARE_PROC_ADDR(ntdll.dll, NTSTATUS, NTAPI, NtFlushBuffersFileEx, + HANDLE FileHandle, ULONG Flags, PVOID Parameters, ULONG ParameterSize, + PIO_STATUS_BLOCK IoStatusBlock); + + if (!INIT_PROC_ADDR(NtFlushBuffersFileEx)) { + errno = ENOSYS; + return -1; + } + + memset(&io_status, 0, sizeof(io_status)); + if (NtFlushBuffersFileEx((HANDLE)_get_osfhandle(fd), FLUSH_FLAGS_FILE_DATA_ONLY, + NULL, 0, &io_status)) { + errno = EINVAL; + return -1; + } + + return 0; +} diff --git a/config.c b/config.c index 2bffa8d4a01..f67f545f839 100644 --- a/config.c +++ b/config.c @@ -1490,6 +1490,18 @@ static int git_default_core_config(const char *var, const char *value, void *cb) return 0; } + if (!strcmp(var, "core.fsyncmethod")) { + if (!value) + return config_error_nonbool(var); + if (!strcmp(value, "fsync")) + fsync_method = FSYNC_METHOD_FSYNC; + else if (!strcmp(value, "writeout-only")) + fsync_method = FSYNC_METHOD_WRITEOUT_ONLY; + else + warning(_("ignoring unknown core.fsyncMethod value '%s'"), value); + + } + if (!strcmp(var, "core.fsyncobjectfiles")) { fsync_object_files = git_config_bool(var, value); return 0; diff --git a/config.mak.uname b/config.mak.uname index c48db45106c..2c67b3b93ce 100644 --- a/config.mak.uname +++ b/config.mak.uname @@ -57,6 +57,7 @@ ifeq ($(uname_S),Linux) HAVE_CLOCK_MONOTONIC = YesPlease # -lrt is needed for clock_gettime on glibc <= 2.16 NEEDS_LIBRT = YesPlease + HAVE_SYNC_FILE_RANGE = YesPlease HAVE_GETDELIM = YesPlease FREAD_READS_DIRECTORIES = UnfortunatelyYes BASIC_CFLAGS += -DHAVE_SYSINFO @@ -462,6 +463,7 @@ endif CFLAGS = BASIC_CFLAGS = -nologo -I. -Icompat/vcbuild/include -DWIN32 -D_CONSOLE -DHAVE_STRING_H -D_CRT_SECURE_NO_WARNINGS -D_CRT_NONSTDC_NO_DEPRECATE COMPAT_OBJS = compat/msvc.o compat/winansi.o \ + compat/win32/flush.o \ compat/win32/path-utils.o \ compat/win32/pthread.o compat/win32/syslog.o \ compat/win32/trace2_win32_process_info.o \ @@ -639,6 +641,7 @@ ifeq ($(uname_S),MINGW) COMPAT_CFLAGS += -DSTRIP_EXTENSION=\".exe\" COMPAT_OBJS += compat/mingw.o compat/winansi.o \ compat/win32/trace2_win32_process_info.o \ + compat/win32/flush.o \ compat/win32/path-utils.o \ compat/win32/pthread.o compat/win32/syslog.o \ compat/win32/dirent.o diff --git a/configure.ac b/configure.ac index d60d494ee4c..660b91f90b4 100644 --- a/configure.ac +++ b/configure.ac @@ -1095,6 +1095,14 @@ AC_COMPILE_IFELSE([CLOCK_MONOTONIC_SRC], [AC_MSG_RESULT([no]) HAVE_CLOCK_MONOTONIC=]) GIT_CONF_SUBST([HAVE_CLOCK_MONOTONIC]) + +# +# Define HAVE_SYNC_FILE_RANGE=YesPlease if sync_file_range is available. +GIT_CHECK_FUNC(sync_file_range, + [HAVE_SYNC_FILE_RANGE=YesPlease], + [HAVE_SYNC_FILE_RANGE]) +GIT_CONF_SUBST([HAVE_SYNC_FILE_RANGE]) + # # Define NO_SETITIMER if you don't have setitimer. GIT_CHECK_FUNC(setitimer, diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt index 5100f56bb37..276e74c1d54 100644 --- a/contrib/buildsystems/CMakeLists.txt +++ b/contrib/buildsystems/CMakeLists.txt @@ -261,7 +261,8 @@ if(CMAKE_SYSTEM_NAME STREQUAL "Windows") NOGDI OBJECT_CREATION_MODE=1 __USE_MINGW_ANSI_STDIO=0 USE_NED_ALLOCATOR OVERRIDE_STRDUP MMAP_PREVENTS_DELETE USE_WIN32_MMAP UNICODE _UNICODE HAVE_WPGMPTR ENSURE_MSYSTEM_IS_SET) - list(APPEND compat_SOURCES compat/mingw.c compat/winansi.c compat/win32/path-utils.c + list(APPEND compat_SOURCES compat/mingw.c compat/winansi.c + compat/win32/flush.c compat/win32/path-utils.c compat/win32/pthread.c compat/win32mmap.c compat/win32/syslog.c compat/win32/trace2_win32_process_info.c compat/win32/dirent.c compat/nedmalloc/nedmalloc.c compat/strdup.c) diff --git a/environment.c b/environment.c index fd0501e77a5..3e3620d759f 100644 --- a/environment.c +++ b/environment.c @@ -44,6 +44,7 @@ int zlib_compression_level = Z_BEST_SPEED; int pack_compression_level = Z_DEFAULT_COMPRESSION; int fsync_object_files; int use_fsync = -1; +enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT; size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE; size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT; size_t delta_base_cache_limit = 96 * 1024 * 1024; diff --git a/git-compat-util.h b/git-compat-util.h index 1229c8296b9..76cb85a56b1 100644 --- a/git-compat-util.h +++ b/git-compat-util.h @@ -1265,6 +1265,30 @@ __attribute__((format (printf, 1, 2))) NORETURN void BUG(const char *fmt, ...); #endif +#ifdef __APPLE__ +#define FSYNC_METHOD_DEFAULT FSYNC_METHOD_WRITEOUT_ONLY +#else +#define FSYNC_METHOD_DEFAULT FSYNC_METHOD_FSYNC +#endif + +enum fsync_action { + FSYNC_WRITEOUT_ONLY, + FSYNC_HARDWARE_FLUSH +}; + +/* + * Issues an fsync against the specified file according to the specified mode. + * + * FSYNC_WRITEOUT_ONLY attempts to use interfaces available on some operating + * systems to flush the OS cache without issuing a flush command to the storage + * controller. If those interfaces are unavailable, the function fails with + * ENOSYS. + * + * FSYNC_HARDWARE_FLUSH does an OS writeout and hardware flush to ensure that + * changes are durable. It is not expected to fail. + */ +int git_fsync(int fd, enum fsync_action action); + /* * Preserves errno, prints a message, but gives no warning for ENOENT. * Returns 0 on success, which includes trying to unlink an object that does diff --git a/wrapper.c b/wrapper.c index 36e12119d76..572f28f14ff 100644 --- a/wrapper.c +++ b/wrapper.c @@ -546,6 +546,70 @@ int xmkstemp_mode(char *filename_template, int mode) return fd; } +/* + * Some platforms return EINTR from fsync. Since fsync is invoked in some + * cases by a wrapper that dies on failure, do not expose EINTR to callers. + */ +static int fsync_loop(int fd) +{ + int err; + + do { + err = fsync(fd); + } while (err < 0 && errno == EINTR); + return err; +} + +int git_fsync(int fd, enum fsync_action action) +{ + switch (action) { + case FSYNC_WRITEOUT_ONLY: + +#ifdef __APPLE__ + /* + * on macOS, fsync just causes filesystem cache writeback but does not + * flush hardware caches. + */ + return fsync_loop(fd); +#endif + +#ifdef HAVE_SYNC_FILE_RANGE + /* + * On linux 2.6.17 and above, sync_file_range is the way to issue + * a writeback without a hardware flush. An offset of 0 and size of 0 + * indicates writeout of the entire file and the wait flags ensure that all + * dirty data is written to the disk (potentially in a disk-side cache) + * before we continue. + */ + + return sync_file_range(fd, 0, 0, SYNC_FILE_RANGE_WAIT_BEFORE | + SYNC_FILE_RANGE_WRITE | + SYNC_FILE_RANGE_WAIT_AFTER); +#endif + +#ifdef fsync_no_flush + return fsync_no_flush(fd); +#endif + + errno = ENOSYS; + return -1; + + case FSYNC_HARDWARE_FLUSH: + /* + * On some platforms fsync may return EINTR. Try again in this + * case, since callers asking for a hardware flush may die if + * this function returns an error. + */ +#ifdef __APPLE__ + return fcntl(fd, F_FULLFSYNC); +#else + return fsync_loop(fd); +#endif + default: + BUG("unexpected git_fsync(%d) call", action); + } +} + static int warn_if_unremovable(const char *op, const char *file, int rc) { int err; diff --git a/write-or-die.c b/write-or-die.c index a3d5784cec9..9faa5f9f563 100644 --- a/write-or-die.c +++ b/write-or-die.c @@ -62,10 +62,13 @@ void fsync_or_die(int fd, const char *msg) use_fsync = git_env_bool("GIT_TEST_FSYNC", 1); if (!use_fsync) return; - while (fsync(fd) < 0) { - if (errno != EINTR) - die_errno("fsync error on '%s'", msg); - } + + if (fsync_method == FSYNC_METHOD_WRITEOUT_ONLY && + git_fsync(fd, FSYNC_WRITEOUT_ONLY) >= 0) + return; + + if (git_fsync(fd, FSYNC_HARDWARE_FLUSH) < 0) + die_errno("fsync error on '%s'", msg); } void write_or_die(int fd, const void *buf, size_t count) -- gitgitgadget ^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v4 2/4] core.fsync: introduce granular fsync control 2022-02-01 3:33 ` [PATCH v4 " Neeraj K. Singh via GitGitGadget 2022-02-01 3:33 ` [PATCH v4 1/4] core.fsyncmethod: add writeout-only mode Neeraj Singh via GitGitGadget @ 2022-02-01 3:33 ` Neeraj Singh via GitGitGadget 2022-02-02 0:51 ` Junio C Hamano 2022-02-01 3:33 ` [PATCH v4 3/4] core.fsync: new option to harden the index Neeraj Singh via GitGitGadget ` (2 subsequent siblings) 4 siblings, 1 reply; 122+ messages in thread From: Neeraj Singh via GitGitGadget @ 2022-02-01 3:33 UTC (permalink / raw) To: git Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps, Neeraj K. Singh, Neeraj Singh From: Neeraj Singh <neerajsi@microsoft.com> This commit introduces the `core.fsync` configuration knob which can be used to control how components of the repository are made durable on disk. This setting allows future extensibility of the list of syncable components: * We issue a warning rather than an error for unrecognized components, so new configs can be used with old Git versions. * We support negation, so users can choose one of the default aggregate options and then remove components that they don't want. The user would then harden any new components added in a Git version update. This also supports the common request of doing absolutely no fysncing with the `core.fsync=none` value, which is expected to make the test suite faster. Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> --- Documentation/config/core.txt | 27 +++++++++---- builtin/fast-import.c | 2 +- builtin/index-pack.c | 4 +- builtin/pack-objects.c | 24 +++++++---- bulk-checkin.c | 5 ++- cache.h | 41 ++++++++++++++++++- commit-graph.c | 3 +- config.c | 76 ++++++++++++++++++++++++++++++++++- csum-file.c | 5 ++- csum-file.h | 3 +- environment.c | 2 +- midx.c | 3 +- object-file.c | 3 +- pack-bitmap-write.c | 3 +- pack-write.c | 13 +++--- read-cache.c | 2 +- 16 files changed, 177 insertions(+), 39 deletions(-) diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt index dbb134f7136..4f1747ec871 100644 --- a/Documentation/config/core.txt +++ b/Documentation/config/core.txt @@ -547,6 +547,25 @@ core.whitespace:: is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent` errors. The default tab width is 8. Allowed values are 1 to 63. +core.fsync:: + A comma-separated list of parts of the repository which should be + hardened via the core.fsyncMethod when created or modified. You can + disable hardening of any component by prefixing it with a '-'. Later + items take precedence over earlier ones in the list. For example, + `core.fsync=all,-pack-metadata` means "harden everything except pack + metadata." Items that are not hardened may be lost in the event of an + unclean system shutdown. ++ +* `none` disables fsync completely. This must be specified alone. +* `loose-object` hardens objects added to the repo in loose-object form. +* `pack` hardens objects added to the repo in packfile form. +* `pack-metadata` hardens packfile bitmaps and indexes. +* `commit-graph` hardens the commit graph file. +* `objects` is an aggregate option that includes `loose-objects`, `pack`, + `pack-metadata`, and `commit-graph`. +* `default` is an aggregate option that is equivalent to `objects,-loose-object` +* `all` is an aggregate option that syncs all individual components above. + core.fsyncMethod:: A value indicating the strategy Git will use to harden repository data using fsync and related primitives. @@ -556,14 +575,6 @@ core.fsyncMethod:: filesystem and storage hardware, data added to the repository may not be durable in the event of a system crash. This is the default mode on macOS. -core.fsyncObjectFiles:: - This boolean will enable 'fsync()' when writing object files. -+ -This is a total waste of time and effort on a filesystem that orders -data writes properly, but can be useful for filesystems that do not use -journalling (traditional UNIX filesystems) or that only journal metadata -and not file contents (OS X's HFS+, or Linux ext3 with "data=writeback"). - core.preloadIndex:: Enable parallel index preload for operations like 'git diff' + diff --git a/builtin/fast-import.c b/builtin/fast-import.c index 2b2e28bad79..ff70aeb1a0e 100644 --- a/builtin/fast-import.c +++ b/builtin/fast-import.c @@ -858,7 +858,7 @@ static void end_packfile(void) struct tag *t; close_pack_windows(pack_data); - finalize_hashfile(pack_file, cur_pack_oid.hash, 0); + finalize_hashfile(pack_file, cur_pack_oid.hash, FSYNC_COMPONENT_PACK, 0); fixup_pack_header_footer(pack_data->pack_fd, pack_data->hash, pack_data->pack_name, object_count, cur_pack_oid.hash, pack_size); diff --git a/builtin/index-pack.c b/builtin/index-pack.c index 3c2e6aee3cc..b871d721e95 100644 --- a/builtin/index-pack.c +++ b/builtin/index-pack.c @@ -1286,7 +1286,7 @@ static void conclude_pack(int fix_thin_pack, const char *curr_pack, unsigned cha nr_objects - nr_objects_initial); stop_progress_msg(&progress, msg.buf); strbuf_release(&msg); - finalize_hashfile(f, tail_hash, 0); + finalize_hashfile(f, tail_hash, FSYNC_COMPONENT_PACK, 0); hashcpy(read_hash, pack_hash); fixup_pack_header_footer(output_fd, pack_hash, curr_pack, nr_objects, @@ -1508,7 +1508,7 @@ static void final(const char *final_pack_name, const char *curr_pack_name, if (!from_stdin) { close(input_fd); } else { - fsync_or_die(output_fd, curr_pack_name); + fsync_component_or_die(FSYNC_COMPONENT_PACK, output_fd, curr_pack_name); err = close(output_fd); if (err) die_errno(_("error while closing pack file")); diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index ba2006f2212..b483ba65adb 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -1199,16 +1199,26 @@ static void write_pack_file(void) display_progress(progress_state, written); } - /* - * Did we write the wrong # entries in the header? - * If so, rewrite it like in fast-import - */ if (pack_to_stdout) { - finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_CLOSE); + /* + * We never fsync when writing to stdout since we may + * not be writing to an actual pack file. For instance, + * the upload-pack code passes a pipe here. Calling + * fsync on a pipe results in unnecessary + * synchronization with the reader on some platforms. + */ + finalize_hashfile(f, hash, FSYNC_COMPONENT_NONE, + CSUM_HASH_IN_STREAM | CSUM_CLOSE); } else if (nr_written == nr_remaining) { - finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); + finalize_hashfile(f, hash, FSYNC_COMPONENT_PACK, + CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); } else { - int fd = finalize_hashfile(f, hash, 0); + /* + * If we wrote the wrong number of entries in the + * header, rewrite it like in fast-import. + */ + + int fd = finalize_hashfile(f, hash, FSYNC_COMPONENT_PACK, 0); fixup_pack_header_footer(fd, hash, pack_tmp_name, nr_written, hash, offset); close(fd); diff --git a/bulk-checkin.c b/bulk-checkin.c index 8785b2ac806..a2cf9dcbc8d 100644 --- a/bulk-checkin.c +++ b/bulk-checkin.c @@ -53,9 +53,10 @@ static void finish_bulk_checkin(struct bulk_checkin_state *state) unlink(state->pack_tmp_name); goto clear_exit; } else if (state->nr_written == 1) { - finalize_hashfile(state->f, hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); + finalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK, + CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); } else { - int fd = finalize_hashfile(state->f, hash, 0); + int fd = finalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK, 0); fixup_pack_header_footer(fd, hash, state->pack_tmp_name, state->nr_written, hash, state->offset); diff --git a/cache.h b/cache.h index 37a32034b2f..b3cd7d928de 100644 --- a/cache.h +++ b/cache.h @@ -993,8 +993,38 @@ void reset_shared_repository(void); extern int read_replace_refs; extern char *git_replace_ref_base; -extern int fsync_object_files; -extern int use_fsync; +/* + * These values are used to help identify parts of a repository to fsync. + * FSYNC_COMPONENT_NONE identifies data that will not be a persistent part of the + * repository and so shouldn't be fsynced. + */ +enum fsync_component { + FSYNC_COMPONENT_NONE, + FSYNC_COMPONENT_LOOSE_OBJECT = 1 << 0, + FSYNC_COMPONENT_PACK = 1 << 1, + FSYNC_COMPONENT_PACK_METADATA = 1 << 2, + FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3, +}; + +#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \ + FSYNC_COMPONENT_PACK_METADATA | \ + FSYNC_COMPONENT_COMMIT_GRAPH) + +#define FSYNC_COMPONENTS_OBJECTS (FSYNC_COMPONENT_LOOSE_OBJECT | \ + FSYNC_COMPONENT_PACK | \ + FSYNC_COMPONENT_PACK_METADATA | \ + FSYNC_COMPONENT_COMMIT_GRAPH) + +#define FSYNC_COMPONENTS_ALL (FSYNC_COMPONENT_LOOSE_OBJECT | \ + FSYNC_COMPONENT_PACK | \ + FSYNC_COMPONENT_PACK_METADATA | \ + FSYNC_COMPONENT_COMMIT_GRAPH) + + +/* + * A bitmask indicating which components of the repo should be fsynced. + */ +extern enum fsync_component fsync_components; enum fsync_method { FSYNC_METHOD_FSYNC, @@ -1002,6 +1032,7 @@ enum fsync_method { }; extern enum fsync_method fsync_method; +extern int use_fsync; extern int core_preload_index; extern int precomposed_unicode; extern int protect_hfs; @@ -1757,6 +1788,12 @@ int copy_file_with_time(const char *dst, const char *src, int mode); void write_or_die(int fd, const void *buf, size_t count); void fsync_or_die(int fd, const char *); +inline void fsync_component_or_die(enum fsync_component component, int fd, const char *msg) +{ + if (fsync_components & component) + fsync_or_die(fd, msg); +} + ssize_t read_in_full(int fd, void *buf, size_t count); ssize_t write_in_full(int fd, const void *buf, size_t count); ssize_t pread_in_full(int fd, void *buf, size_t count, off_t offset); diff --git a/commit-graph.c b/commit-graph.c index 265c010122e..64897f57d9f 100644 --- a/commit-graph.c +++ b/commit-graph.c @@ -1942,7 +1942,8 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx) } close_commit_graph(ctx->r->objects); - finalize_hashfile(f, file_hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC); + finalize_hashfile(f, file_hash, FSYNC_COMPONENT_COMMIT_GRAPH, + CSUM_HASH_IN_STREAM | CSUM_FSYNC); free_chunkfile(cf); if (ctx->split) { diff --git a/config.c b/config.c index f67f545f839..224563c7b3e 100644 --- a/config.c +++ b/config.c @@ -1213,6 +1213,73 @@ static int git_parse_maybe_bool_text(const char *value) return -1; } +static const struct fsync_component_entry { + const char *name; + enum fsync_component component_bits; +} fsync_component_table[] = { + { "loose-object", FSYNC_COMPONENT_LOOSE_OBJECT }, + { "pack", FSYNC_COMPONENT_PACK }, + { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA }, + { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, + { "objects", FSYNC_COMPONENTS_OBJECTS }, + { "default", FSYNC_COMPONENTS_DEFAULT }, + { "all", FSYNC_COMPONENTS_ALL }, +}; + +static enum fsync_component parse_fsync_components(const char *var, const char *string) +{ + enum fsync_component output = 0; + + if (!strcmp(string, "none")) + return output; + + while (string) { + int i; + size_t len; + const char *ep; + int negated = 0; + int found = 0; + + string = string + strspn(string, ", \t\n\r"); + ep = strchrnul(string, ','); + len = ep - string; + + if (*string == '-') { + negated = 1; + string++; + len--; + if (!len) + warning(_("invalid value for variable %s"), var); + } + + if (!len) + break; + + for (i = 0; i < ARRAY_SIZE(fsync_component_table); ++i) { + const struct fsync_component_entry *entry = &fsync_component_table[i]; + + if (strncmp(entry->name, string, len)) + continue; + + found = 1; + if (negated) + output &= ~entry->component_bits; + else + output |= entry->component_bits; + } + + if (!found) { + char *component = xstrndup(string, len); + warning(_("ignoring unknown core.fsync component '%s'"), component); + free(component); + } + + string = ep; + } + + return output; +} + int git_parse_maybe_bool(const char *value) { int v = git_parse_maybe_bool_text(value); @@ -1490,6 +1557,13 @@ static int git_default_core_config(const char *var, const char *value, void *cb) return 0; } + if (!strcmp(var, "core.fsync")) { + if (!value) + return config_error_nonbool(var); + fsync_components = parse_fsync_components(var, value); + return 0; + } + if (!strcmp(var, "core.fsyncmethod")) { if (!value) return config_error_nonbool(var); @@ -1503,7 +1577,7 @@ static int git_default_core_config(const char *var, const char *value, void *cb) } if (!strcmp(var, "core.fsyncobjectfiles")) { - fsync_object_files = git_config_bool(var, value); + warning(_("core.fsyncobjectfiles is deprecated; use core.fsync instead")); return 0; } diff --git a/csum-file.c b/csum-file.c index 26e8a6df44e..59ef3398ca2 100644 --- a/csum-file.c +++ b/csum-file.c @@ -58,7 +58,8 @@ static void free_hashfile(struct hashfile *f) free(f); } -int finalize_hashfile(struct hashfile *f, unsigned char *result, unsigned int flags) +int finalize_hashfile(struct hashfile *f, unsigned char *result, + enum fsync_component component, unsigned int flags) { int fd; @@ -69,7 +70,7 @@ int finalize_hashfile(struct hashfile *f, unsigned char *result, unsigned int fl if (flags & CSUM_HASH_IN_STREAM) flush(f, f->buffer, the_hash_algo->rawsz); if (flags & CSUM_FSYNC) - fsync_or_die(f->fd, f->name); + fsync_component_or_die(component, f->fd, f->name); if (flags & CSUM_CLOSE) { if (close(f->fd)) die_errno("%s: sha1 file error on close", f->name); diff --git a/csum-file.h b/csum-file.h index 291215b34eb..0d29f528fbc 100644 --- a/csum-file.h +++ b/csum-file.h @@ -1,6 +1,7 @@ #ifndef CSUM_FILE_H #define CSUM_FILE_H +#include "cache.h" #include "hash.h" struct progress; @@ -38,7 +39,7 @@ int hashfile_truncate(struct hashfile *, struct hashfile_checkpoint *); struct hashfile *hashfd(int fd, const char *name); struct hashfile *hashfd_check(const char *name); struct hashfile *hashfd_throughput(int fd, const char *name, struct progress *tp); -int finalize_hashfile(struct hashfile *, unsigned char *, unsigned int); +int finalize_hashfile(struct hashfile *, unsigned char *, enum fsync_component, unsigned int); void hashwrite(struct hashfile *, const void *, unsigned int); void hashflush(struct hashfile *f); void crc32_begin(struct hashfile *); diff --git a/environment.c b/environment.c index 3e3620d759f..378424b9af5 100644 --- a/environment.c +++ b/environment.c @@ -42,9 +42,9 @@ const char *git_attributes_file; const char *git_hooks_path; int zlib_compression_level = Z_BEST_SPEED; int pack_compression_level = Z_DEFAULT_COMPRESSION; -int fsync_object_files; int use_fsync = -1; enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT; +enum fsync_component fsync_components = FSYNC_COMPONENTS_DEFAULT; size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE; size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT; size_t delta_base_cache_limit = 96 * 1024 * 1024; diff --git a/midx.c b/midx.c index 837b46b2af5..882f91f7d57 100644 --- a/midx.c +++ b/midx.c @@ -1406,7 +1406,8 @@ static int write_midx_internal(const char *object_dir, write_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs); write_chunkfile(cf, &ctx); - finalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM); + finalize_hashfile(f, midx_hash, FSYNC_COMPONENT_PACK_METADATA, + CSUM_FSYNC | CSUM_HASH_IN_STREAM); free_chunkfile(cf); if (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP)) diff --git a/object-file.c b/object-file.c index 8be57f48de7..ff29e678204 100644 --- a/object-file.c +++ b/object-file.c @@ -1850,8 +1850,7 @@ int hash_object_file(const struct git_hash_algo *algo, const void *buf, static void close_loose_object(int fd) { if (!the_repository->objects->odb->will_destroy) { - if (fsync_object_files) - fsync_or_die(fd, "loose object file"); + fsync_component_or_die(FSYNC_COMPONENT_LOOSE_OBJECT, fd, "loose object file"); } if (close(fd) != 0) diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c index 9c55c1531e1..c16e43d1669 100644 --- a/pack-bitmap-write.c +++ b/pack-bitmap-write.c @@ -719,7 +719,8 @@ void bitmap_writer_finish(struct pack_idx_entry **index, if (options & BITMAP_OPT_HASH_CACHE) write_hash_cache(f, index, index_nr); - finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); + finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA, + CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); if (adjust_shared_perm(tmp_file.buf)) die_errno("unable to make temporary bitmap file readable"); diff --git a/pack-write.c b/pack-write.c index a5846f3a346..51812cb1299 100644 --- a/pack-write.c +++ b/pack-write.c @@ -159,9 +159,9 @@ const char *write_idx_file(const char *index_name, struct pack_idx_entry **objec } hashwrite(f, sha1, the_hash_algo->rawsz); - finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE | - ((opts->flags & WRITE_IDX_VERIFY) - ? 0 : CSUM_FSYNC)); + finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA, + CSUM_HASH_IN_STREAM | CSUM_CLOSE | + ((opts->flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC)); return index_name; } @@ -281,8 +281,9 @@ const char *write_rev_file_order(const char *rev_name, if (rev_name && adjust_shared_perm(rev_name) < 0) die(_("failed to make %s readable"), rev_name); - finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE | - ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC)); + finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA, + CSUM_HASH_IN_STREAM | CSUM_CLOSE | + ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC)); return rev_name; } @@ -390,7 +391,7 @@ void fixup_pack_header_footer(int pack_fd, the_hash_algo->final_fn(partial_pack_hash, &old_hash_ctx); the_hash_algo->final_fn(new_pack_hash, &new_hash_ctx); write_or_die(pack_fd, new_pack_hash, the_hash_algo->rawsz); - fsync_or_die(pack_fd, pack_name); + fsync_component_or_die(FSYNC_COMPONENT_PACK, pack_fd, pack_name); } char *index_pack_lockfile(int ip_out, int *is_well_formed) diff --git a/read-cache.c b/read-cache.c index cbe73f14e5e..a0de70195c8 100644 --- a/read-cache.c +++ b/read-cache.c @@ -3081,7 +3081,7 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile, return -1; } - finalize_hashfile(f, istate->oid.hash, CSUM_HASH_IN_STREAM); + finalize_hashfile(f, istate->oid.hash, FSYNC_COMPONENT_NONE, CSUM_HASH_IN_STREAM); if (close_tempfile_gently(tempfile)) { error(_("could not close '%s'"), get_tempfile_path(tempfile)); return -1; -- gitgitgadget ^ permalink raw reply related [flat|nested] 122+ messages in thread
* Re: [PATCH v4 2/4] core.fsync: introduce granular fsync control 2022-02-01 3:33 ` [PATCH v4 2/4] core.fsync: introduce granular fsync control Neeraj Singh via GitGitGadget @ 2022-02-02 0:51 ` Junio C Hamano 2022-02-02 1:42 ` Junio C Hamano 2022-02-11 20:38 ` Neeraj Singh 0 siblings, 2 replies; 122+ messages in thread From: Junio C Hamano @ 2022-02-02 0:51 UTC (permalink / raw) To: Neeraj Singh via GitGitGadget Cc: git, rsbecker, bagasdotme, newren, avarab, nksingh85, ps, Neeraj K. Singh "Neeraj Singh via GitGitGadget" <gitgitgadget@gmail.com> writes: > +core.fsync:: > + A comma-separated list of parts of the repository which should be > + hardened via the core.fsyncMethod when created or modified. You can > + disable hardening of any component by prefixing it with a '-'. Later > + items take precedence over earlier ones in the list. For example, > + `core.fsync=all,-pack-metadata` means "harden everything except pack > + metadata." Items that are not hardened may be lost in the event of an > + unclean system shutdown. > ++ > +* `none` disables fsync completely. This must be specified alone. > +* `loose-object` hardens objects added to the repo in loose-object form. > +* `pack` hardens objects added to the repo in packfile form. > +* `pack-metadata` hardens packfile bitmaps and indexes. > +* `commit-graph` hardens the commit graph file. > +* `objects` is an aggregate option that includes `loose-objects`, `pack`, > + `pack-metadata`, and `commit-graph`. > +* `default` is an aggregate option that is equivalent to `objects,-loose-object` > +* `all` is an aggregate option that syncs all individual components above. I am not quite sure if this is way too complex (e.g. what does it mean that we do not care much about loose-object safety while we do care about commit-graph files?) and at the same time it is too limited (e.g. if it makes sense to say a class of items deserve more protection than another class of items, don't we want to be able to say "class X is ultra-precious so use method A on them, while class Y is mildly precious and use method B on them, everything else are not that important and doing the default thing is just fine"). If we wanted to allow the "matrix" kind of flexibility, I think the way to do so would be fsync.<class>.method = <value> e.g. [fsync "default"] method = none [fsync "loose-object"] method = fsync [fsync "pack-metadata"] method = writeout-only Where do we expect users to take the core.fsync settings from? Per repository? If it is from per user (i.e. $HOME/.gitconfig), do people tend to share it across systems (not necessarily over NFS) with the same contents? If so, I am not sure if fsync.method that is way too close to the actual "implementation" is a good idea to begin with. From end-user's point of view, it may be easier to express "class X is ultra-precious, and class Y and Z are mildly so", with something like fsync.<class>.level = <how-precious> and let the Git implementation on each platform choose the appropriate fsync method to protect the stuff at that precious-ness. Thanks. ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v4 2/4] core.fsync: introduce granular fsync control 2022-02-02 0:51 ` Junio C Hamano @ 2022-02-02 1:42 ` Junio C Hamano 2022-02-11 21:18 ` Neeraj Singh 2022-02-11 20:38 ` Neeraj Singh 1 sibling, 1 reply; 122+ messages in thread From: Junio C Hamano @ 2022-02-02 1:42 UTC (permalink / raw) To: Neeraj Singh via GitGitGadget Cc: git, rsbecker, bagasdotme, newren, avarab, nksingh85, ps, Neeraj K. Singh Junio C Hamano <gitster@pobox.com> writes: > I am not quite sure if this is way too complex (e.g. what does it > mean that we do not care much about loose-object safety while we do > care about commit-graph files?) and at the same time it is too > limited (e.g. if it makes sense to say a class of items deserve more > protection than another class of items, don't we want to be able to > say "class X is ultra-precious so use method A on them, while class > Y is mildly precious and use method B on them, everything else are > not that important and doing the default thing is just fine"). > > If we wanted to allow the "matrix" kind of flexibility,... To continue with the thinking aloud... Sometimes configuration flexibility is truly needed, but often it is just a sign of designer being lazy and not thinking it through as an end-user facing problem. In other words, "I am giving enough knobs to you, so it is up to you to express your policy in whatever way you want with the knobs provided" is a very irresponsible thing to tell end-users. And this one smells like the case of a lazy design. It may be that it makes sense in some workflows to protect commit-graph files less than object files and pack.idx files can be corrupted as long as pack.pack files are adequately protected because the former can be recomputed from the latter, but in no workflows, the reverse would be true. Yet the design gives such needless flexibility, which makes it hard for lay end-users to choose the best combination and allows them to protect .idx files more than .pack files by mistake, for example. I am wondering if the classification itself introduced by this step actually can form a natural and linear progression of safe-ness. By default, we'd want _all_ classes of things to be equally safe, but at one level down, there is "protect things that are not recomputable, but recomputable things can be left to the system" level, and there would be even riskier "protect packs as it would hurt a _lot_ to lose them, but losing loose ones will typically lose only the most recent work, and they are less valuable" level. If we, as the Git experts, spend extra brain cycles to come up with an easy to understand spectrum of performance vs durability trade-off, end-users won't have to learn the full flexibility and easily take the advice from experts. They just need to say what level of durability they want (or how much durability they can risk in exchange for an additional throughput), and leave the rest to us. On the core.fsyncMethod side, the same suggestion applies. Once we know the desired level of performance vs durability trade-off from the user, we, as the impolementors, should know the best method, for each class of items, to achieve that durability on each platform when writing it to the storage, without exposing the low level details of the implementation that only the Git folks need to be aware of. So, from the end-user UI perspective, I'd very much prefer if we can just come up with a single scalar variable, (say "fsync.durability" that ranges from "conservative" to "performance") that lets our users express the level of durability desired. The combination of core.fsyncMethod and core.fsync are one variable too many, and the latter being a variable that takes a list of things as its value makes it even worse to sell to the end users. ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v4 2/4] core.fsync: introduce granular fsync control 2022-02-02 1:42 ` Junio C Hamano @ 2022-02-11 21:18 ` Neeraj Singh 2022-02-11 22:19 ` Junio C Hamano 0 siblings, 1 reply; 122+ messages in thread From: Neeraj Singh @ 2022-02-11 21:18 UTC (permalink / raw) To: Junio C Hamano Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Ævar Arnfjörð Bjarmason, Patrick Steinhardt, Neeraj K. Singh On Tue, Feb 1, 2022 at 5:42 PM Junio C Hamano <gitster@pobox.com> wrote: > > Junio C Hamano <gitster@pobox.com> writes: > > > I am not quite sure if this is way too complex (e.g. what does it > > mean that we do not care much about loose-object safety while we do > > care about commit-graph files?) and at the same time it is too > > limited (e.g. if it makes sense to say a class of items deserve more > > protection than another class of items, don't we want to be able to > > say "class X is ultra-precious so use method A on them, while class > > Y is mildly precious and use method B on them, everything else are > > not that important and doing the default thing is just fine"). > > > > If we wanted to allow the "matrix" kind of flexibility,... > > To continue with the thinking aloud... > > Sometimes configuration flexibility is truly needed, but often it is > just a sign of designer being lazy and not thinking it through as an > end-user facing problem. In other words, "I am giving enough knobs > to you, so it is up to you to express your policy in whatever way > you want with the knobs provided" is a very irresponsible thing to > tell end-users. > > And this one smells like the case of a lazy design. > > It may be that it makes sense in some workflows to protect > commit-graph files less than object files and pack.idx files can be > corrupted as long as pack.pack files are adequately protected > because the former can be recomputed from the latter, but in no > workflows, the reverse would be true. Yet the design gives such > needless flexibility, which makes it hard for lay end-users to > choose the best combination and allows them to protect .idx files > more than .pack files by mistake, for example. > > I am wondering if the classification itself introduced by this step > actually can form a natural and linear progression of safe-ness. By > default, we'd want _all_ classes of things to be equally safe, but > at one level down, there is "protect things that are not > recomputable, but recomputable things can be left to the system" > level, and there would be even riskier "protect packs as it would > hurt a _lot_ to lose them, but losing loose ones will typically lose > only the most recent work, and they are less valuable" level. > > If we, as the Git experts, spend extra brain cycles to come up with > an easy to understand spectrum of performance vs durability > trade-off, end-users won't have to learn the full flexibility and > easily take the advice from experts. They just need to say what > level of durability they want (or how much durability they can risk > in exchange for an additional throughput), and leave the rest to us. > > On the core.fsyncMethod side, the same suggestion applies. > > Once we know the desired level of performance vs durability > trade-off from the user, we, as the impolementors, should know the > best method, for each class of items, to achieve that durability on > each platform when writing it to the storage, without exposing the > low level details of the implementation that only the Git folks need > to be aware of. > > So, from the end-user UI perspective, I'd very much prefer if we can > just come up with a single scalar variable, (say "fsync.durability" > that ranges from "conservative" to "performance") that lets our > users express the level of durability desired. The combination of > core.fsyncMethod and core.fsync are one variable too many, and the > latter being a variable that takes a list of things as its value > makes it even worse to sell to the end users. I see the value in simplifying the core.fsync configuration to a single scalar knob of preciousness. The main motivation for this more granular scheme is that I didn't think the current configuration follows a sensible principle. We should be fsyncing the loose objects, index, refs, and config files in addition to what we're already syncing today. On macOS, we should be doing a full hardware flush if we've said we want to fsync. But you expressed the notion in [1] that we don't want to degrade the performance of the vast majority of users who are happy with the current "unprincipled but mostly works" configuration. I agree with that sentiment, but it leads to a design where we can express the current configuration, which does not follow a scalar hierarchy. The aggregate core.fsync options are meant to provide a way for us to recommend a sensible configuration to the user without having to get into the intricacies of repo layout. Maybe we can define and document aggregate options that make sense in terms of a scalar level of preciousness. One reason to keep core.fsyncMethod separate from the core.fsync knob is that it's more a property of the system and FS the repo is on, rather than the user's value of the repo. We could try to auto-detect some known filesystems that might support batch mode using 'statfs', but having a table like that in Git would go out of date over time. Please let me know what you think about these justifications for the current design. I'd be happy to make a change if the current constraint of "keep the default config the same" can be relaxed in some way. I'd also be happy to go back to some variation of expressing 'core.fsyncObjectFiles = batch' and leaving the rest of fsync story alone. Thanks, Neeraj [1] https://lore.kernel.org/git/xmqqtuilyfls.fsf@gitster.g/ ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v4 2/4] core.fsync: introduce granular fsync control 2022-02-11 21:18 ` Neeraj Singh @ 2022-02-11 22:19 ` Junio C Hamano 2022-02-11 23:04 ` Neeraj Singh 0 siblings, 1 reply; 122+ messages in thread From: Junio C Hamano @ 2022-02-11 22:19 UTC (permalink / raw) To: Neeraj Singh Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Ævar Arnfjörð Bjarmason, Patrick Steinhardt, Neeraj K. Singh Neeraj Singh <nksingh85@gmail.com> writes: > The main motivation for this more granular scheme is that I didn't > think the current configuration follows a sensible principle. We > should be fsyncing the loose objects, index, refs, and config > files in addition to what we're already syncing today. On macOS, > we should be doing a full hardware flush if we've said we want to > fsync. If the "robustness vs performance" trade-off is unevenly made in the current code, then that is a very good problem to address first, and such a change is very much justified on its own. Perhaps "this is not a primary work repository but is used only to follow external site to build, hence no fsync is fine" folks, who do not have core.fsyncObjectFiles set to true, may appreciate if we stopped doing fsync for packs and other things. As the Boolean core.fsyncObjectFiles is the only end-user visible knob to express how the end-users express the trade-off, a good first step would be to align other file accesses to the preference expressed by it, i.e. others who say they want fsync in the current scheme would appreciate if we start fsync in places like ref-files backend. Making the choice more granular, from Boolean "yes/no", to linear levels, would also be a good idea. Doing both at the same time may make it harder to explain and justify, but as long as at the end, if "very performant" choice uniformly does not do any fsync while "ultra durable" choice makes a uniform effort across subsystems to make sure bits hit the platter, it would be a very good idea to do them. Thanks. ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v4 2/4] core.fsync: introduce granular fsync control 2022-02-11 22:19 ` Junio C Hamano @ 2022-02-11 23:04 ` Neeraj Singh 2022-02-11 23:15 ` Junio C Hamano 0 siblings, 1 reply; 122+ messages in thread From: Neeraj Singh @ 2022-02-11 23:04 UTC (permalink / raw) To: Junio C Hamano Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Ævar Arnfjörð Bjarmason, Patrick Steinhardt, Neeraj K. Singh On Fri, Feb 11, 2022 at 2:19 PM Junio C Hamano <gitster@pobox.com> wrote: > > Neeraj Singh <nksingh85@gmail.com> writes: > > > The main motivation for this more granular scheme is that I didn't > > think the current configuration follows a sensible principle. We > > should be fsyncing the loose objects, index, refs, and config > > files in addition to what we're already syncing today. On macOS, > > we should be doing a full hardware flush if we've said we want to > > fsync. > > If the "robustness vs performance" trade-off is unevenly made in the > current code, then that is a very good problem to address first, and > such a change is very much justified on its own. > > Perhaps "this is not a primary work repository but is used only to > follow external site to build, hence no fsync is fine" folks, who do > not have core.fsyncObjectFiles set to true, may appreciate if we > stopped doing fsync for packs and other things. As the Boolean > core.fsyncObjectFiles is the only end-user visible knob to express > how the end-users express the trade-off, a good first step would be > to align other file accesses to the preference expressed by it, i.e. > others who say they want fsync in the current scheme would > appreciate if we start fsync in places like ref-files backend. > In practice, almost all users have core.fsyncObjectFiles set to the platform default, which is 'false' everywhere besides Windows. So at minimum, we have to take default to mean that we maintain behavior no weaker than the current version of Git, otherwise users will start losing their data. The advantage of introducing a new knob is that we don't have to try to divine why the user set a particular value of core.fsyncObjectFiles. > Making the choice more granular, from Boolean "yes/no", to linear > levels, would also be a good idea. Doing both at the same time may > make it harder to explain and justify, but as long as at the end, if > "very performant" choice uniformly does not do any fsync while > "ultra durable" choice makes a uniform effort across subsystems to > make sure bits hit the platter, it would be a very good idea to do > them. > One path to get to your suggestion from the current patch series would be to remove the component-specific options and only provide aggregate options. Alternatively, we could just not document the component-specific options and leave them available to be people who read source code. So if I rename the aggregate options in terms of 'levels of durability', and only document those, would that be acceptable? Thanks for the review! -Neeraj ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v4 2/4] core.fsync: introduce granular fsync control 2022-02-11 23:04 ` Neeraj Singh @ 2022-02-11 23:15 ` Junio C Hamano 2022-02-12 0:39 ` rsbecker 2022-02-14 7:04 ` Patrick Steinhardt 0 siblings, 2 replies; 122+ messages in thread From: Junio C Hamano @ 2022-02-11 23:15 UTC (permalink / raw) To: Neeraj Singh Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Ævar Arnfjörð Bjarmason, Patrick Steinhardt, Neeraj K. Singh Neeraj Singh <nksingh85@gmail.com> writes: > In practice, almost all users have core.fsyncObjectFiles set to the > platform default, which is 'false' everywhere besides Windows. So at > minimum, we have to take default to mean that we maintain behavior no > weaker than the current version of Git, otherwise users will start > losing their data. Would they? If they think platform default is performant and safe enough for their use, as long as our adjustment is out outrageously more dangerous or less performant, I do not think "no weaker than" is a strict requirement. If we were overly conservative in some areas than the "platform default", making it less conservative in those areas to match the looseness of other areas should be OK and vice versa. > One path to get to your suggestion from the current patch series would > be to remove the component-specific options and only provide aggregate > options. Alternatively, we could just not document the > component-specific options and leave them available to be people who > read source code. So if I rename the aggregate options in terms of > 'levels of durability', and only document those, would that be > acceptable? In any case, if others who reviewed the series in the past are happy with the "two knobs" approach and are willing to jump in to help new users who will be confused with one knob too many, I actually am OK with the series that I called "overly complex". Let me let them weigh in before I can answer that question. Thanks. ^ permalink raw reply [flat|nested] 122+ messages in thread
* RE: [PATCH v4 2/4] core.fsync: introduce granular fsync control 2022-02-11 23:15 ` Junio C Hamano @ 2022-02-12 0:39 ` rsbecker 2022-02-14 7:04 ` Patrick Steinhardt 1 sibling, 0 replies; 122+ messages in thread From: rsbecker @ 2022-02-12 0:39 UTC (permalink / raw) To: 'Junio C Hamano', 'Neeraj Singh' Cc: 'Neeraj Singh via GitGitGadget', 'Git List', 'Bagas Sanjaya', 'Elijah Newren', 'Ævar Arnfjörð Bjarmason', 'Patrick Steinhardt', 'Neeraj K. Singh' On February 11, 2022 6:15 PM, Junio C Hamano wrote: > Neeraj Singh <nksingh85@gmail.com> writes: > > > In practice, almost all users have core.fsyncObjectFiles set to the > > platform default, which is 'false' everywhere besides Windows. So at > > minimum, we have to take default to mean that we maintain behavior no > > weaker than the current version of Git, otherwise users will start > > losing their data. > > Would they? If they think platform default is performant and safe > enough for their use, as long as our adjustment is out outrageously more > dangerous or less performant, I do not think "no weaker than" > is a strict requirement. If we were overly conservative in some areas than the > "platform default", making it less conservative in those areas to match the > looseness of other areas should be OK and vice versa. > > > One path to get to your suggestion from the current patch series would > > be to remove the component-specific options and only provide aggregate > > options. Alternatively, we could just not document the > > component-specific options and leave them available to be people who > > read source code. So if I rename the aggregate options in terms of > > 'levels of durability', and only document those, would that be > > acceptable? > > In any case, if others who reviewed the series in the past are happy with the > "two knobs" approach and are willing to jump in to help new users who will be > confused with one knob too many, I actually am OK with the series that I called > "overly complex". Let me let them weigh in before I can answer that question. On behalf of those who are likely to set fsync to true in all cases, because a SIGSEGV or some other early abort will cause changes to be lost, I am not happy with excessive knobs, as we will have to ensure that the are all set to "write this out to disk as quickly as soon as possible or else". I end up having to teach large numbers of people about these settings and think that excessive controls in this area are overwhelmingly bad. This is not a "windows vs. everything else" situation. Not all platforms write buffered by default. fwrite and write behave different on NonStop (and other POSIXy things, and some variants are fully virtualized, so who knows what the hypervisor will do - it's bad enough that there is even an option to tell the hypervisor to keep things in memory until convenient to write even on RAID 0/1 emulation). Teaching people how to use the knobs correctly is going to be a challenge. For me, I'm always willing to sacrifice performance for reliability - 100% of the time. I'm sure this whole series is going to be problematic. I'm sorry but that's my position on it. The default position of all knobs must be to force the write and keeping it simple so that variations on knob settings give different results is not going to end well. Sincerely, Randall ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v4 2/4] core.fsync: introduce granular fsync control 2022-02-11 23:15 ` Junio C Hamano 2022-02-12 0:39 ` rsbecker @ 2022-02-14 7:04 ` Patrick Steinhardt 2022-02-14 17:17 ` Junio C Hamano 1 sibling, 1 reply; 122+ messages in thread From: Patrick Steinhardt @ 2022-02-14 7:04 UTC (permalink / raw) To: Junio C Hamano Cc: Neeraj Singh, Neeraj Singh via GitGitGadget, Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Ævar Arnfjörð Bjarmason, Neeraj K. Singh [-- Attachment #1: Type: text/plain, Size: 4425 bytes --] On Fri, Feb 11, 2022 at 03:15:20PM -0800, Junio C Hamano wrote: > Neeraj Singh <nksingh85@gmail.com> writes: > > > In practice, almost all users have core.fsyncObjectFiles set to the > > platform default, which is 'false' everywhere besides Windows. So at > > minimum, we have to take default to mean that we maintain behavior no > > weaker than the current version of Git, otherwise users will start > > losing their data. > > Would they? If they think platform default is performant and safe > enough for their use, as long as our adjustment is out outrageously > more dangerous or less performant, I do not think "no weaker than" > is a strict requirement. If we were overly conservative in some > areas than the "platform default", making it less conservative in > those areas to match the looseness of other areas should be OK and > vice versa. > > > One path to get to your suggestion from the current patch series would > > be to remove the component-specific options and only provide aggregate > > options. Alternatively, we could just not document the > > component-specific options and leave them available to be people who > > read source code. So if I rename the aggregate options in terms of > > 'levels of durability', and only document those, would that be > > acceptable? > > In any case, if others who reviewed the series in the past are happy > with the "two knobs" approach and are willing to jump in to help new > users who will be confused with one knob too many, I actually am OK > with the series that I called "overly complex". Let me let them > weigh in before I can answer that question. > > Thanks. I wonder whether it makes sense to distinguish client- and server-side requirements. While we probably want to "do the right thing" on the client-side by default so that we don't have to teach users that "We may use data in some cases on some systems, but not in other cases on other systems by default." On the server-side folks who implement the Git hosting are typically a lot more knowledgeable with regards to how Git behaves, and it can be expected of them to dig a lot deeper than we can and should reasonably expect from a user. One point I really care about and that I think is extremely important in the context of both client and server is data consistency. While it is bad to lose data, the reality is that it can simply happen when systems hit exceptional cases like a hard-reset. But what we really must not let happen is that as a result, a repository gets corrupted because of this hard reset. In the context of client-side repositories a user will be left to wonder how to fix the repository, while on the server-side we need to go in and somehow repair the repository fast or otherwise users will come to us and complain their repository stopped working. Our current defaults can end up with repository corruption though. We don't sync object files before renaming them into place by default, and furthermore we don't even give a knob to fsync loose references before renaming them. On gitlab.com we enable "core.fsyncObjectFiles" and thus to the best of my knowledge never hit a corrupted ODB until now. But what we do regularly hit is corrupted references because there is no knob to fsync them. Furthermore, I really think that the advice in git-config(1) with regards to loose syncing loose objects that "This is a total waste of time and effort" is wrong. Filesystem developers have repeatedly stated that for proper atomicity guarantees, a file must be synced to disk before renaming it into place [1][2][3]. So from the perspective of the people who write Linux-based filesystems our application is "broken" right now, even though there are mechanisms in place which try to work around our brokenness by using heuristics to detect what we're doing. To summarize my take: while the degree of durability may be something that's up for discussions, I think that the current defaults for atomicity are bad for users because they can and do lead to repository corruption. Patrick [1]: https://thunk.org/tytso/blog/2009/03/15/dont-fear-the-fsync/ [2]: https://btrfs.wiki.kernel.org/index.php/FAQ (What are the crash guarantees of overwrite-by-rename) [3]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/admin-guide/ext4.rst (see auto_da_alloc) [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v4 2/4] core.fsync: introduce granular fsync control 2022-02-14 7:04 ` Patrick Steinhardt @ 2022-02-14 17:17 ` Junio C Hamano 2022-03-09 13:42 ` Patrick Steinhardt 0 siblings, 1 reply; 122+ messages in thread From: Junio C Hamano @ 2022-02-14 17:17 UTC (permalink / raw) To: Patrick Steinhardt Cc: Neeraj Singh, Neeraj Singh via GitGitGadget, Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Ævar Arnfjörð Bjarmason, Neeraj K. Singh Patrick Steinhardt <ps@pks.im> writes: > To summarize my take: while the degree of durability may be something > that's up for discussions, I think that the current defaults for > atomicity are bad for users because they can and do lead to repository > corruption. Good summary. If the user cares about fsynching loose object files in the right way, we shouldn't leave loose ref files not following the safe safety level, regardless of how this new core.fsync knobs would look like. I think we three are in agreement on that. ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v4 2/4] core.fsync: introduce granular fsync control 2022-02-14 17:17 ` Junio C Hamano @ 2022-03-09 13:42 ` Patrick Steinhardt 2022-03-09 18:50 ` Ævar Arnfjörð Bjarmason ` (2 more replies) 0 siblings, 3 replies; 122+ messages in thread From: Patrick Steinhardt @ 2022-03-09 13:42 UTC (permalink / raw) To: Junio C Hamano Cc: Neeraj Singh, Neeraj Singh via GitGitGadget, Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Ævar Arnfjörð Bjarmason, Neeraj K. Singh [-- Attachment #1: Type: text/plain, Size: 1429 bytes --] On Mon, Feb 14, 2022 at 09:17:31AM -0800, Junio C Hamano wrote: > Patrick Steinhardt <ps@pks.im> writes: > > > To summarize my take: while the degree of durability may be something > > that's up for discussions, I think that the current defaults for > > atomicity are bad for users because they can and do lead to repository > > corruption. > > Good summary. > > If the user cares about fsynching loose object files in the right > way, we shouldn't leave loose ref files not following the safe > safety level, regardless of how this new core.fsync knobs would look > like. > > I think we three are in agreement on that. Is there anything I can specifically do to help out with this topic? We have again hit data loss in production because we don't sync loose refs to disk before renaming them into place, so I'd really love to sort out this issue somehow so that I can revive my patch series which fixes the known repository corruption [1]. Alternatively, can we maybe find a way forward with applying a version of my patch series without first settling the bigger question of how we want the overall design to look like? In my opinion repository corruption is a severe bug that needs to be fixed, and it doesn't feel sensible to block such a fix over a discussion that potentially will take a long time to settle. Patrick [1]: http://public-inbox.org/git/cover.1636544377.git.ps@pks.im/ [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v4 2/4] core.fsync: introduce granular fsync control 2022-03-09 13:42 ` Patrick Steinhardt @ 2022-03-09 18:50 ` Ævar Arnfjörð Bjarmason 2022-03-09 20:03 ` Junio C Hamano 2022-03-09 20:05 ` Neeraj Singh 2 siblings, 0 replies; 122+ messages in thread From: Ævar Arnfjörð Bjarmason @ 2022-03-09 18:50 UTC (permalink / raw) To: Patrick Steinhardt Cc: Junio C Hamano, Neeraj Singh, Neeraj Singh via GitGitGadget, Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Neeraj K. Singh On Wed, Mar 09 2022, Patrick Steinhardt wrote: > [[PGP Signed Part:Undecided]] > On Mon, Feb 14, 2022 at 09:17:31AM -0800, Junio C Hamano wrote: >> Patrick Steinhardt <ps@pks.im> writes: >> >> > To summarize my take: while the degree of durability may be something >> > that's up for discussions, I think that the current defaults for >> > atomicity are bad for users because they can and do lead to repository >> > corruption. >> >> Good summary. >> >> If the user cares about fsynching loose object files in the right >> way, we shouldn't leave loose ref files not following the safe >> safety level, regardless of how this new core.fsync knobs would look >> like. >> >> I think we three are in agreement on that. > > Is there anything I can specifically do to help out with this topic? We > have again hit data loss in production because we don't sync loose refs > to disk before renaming them into place, so I'd really love to sort out > this issue somehow so that I can revive my patch series which fixes the > known repository corruption [1]. > > Alternatively, can we maybe find a way forward with applying a version > of my patch series without first settling the bigger question of how we > want the overall design to look like? In my opinion repository > corruption is a severe bug that needs to be fixed, and it doesn't feel > sensible to block such a fix over a discussion that potentially will > take a long time to settle. > > Patrick > > [1]: http://public-inbox.org/git/cover.1636544377.git.ps@pks.im/ I share that view. I was wondering how this topic fizzled out the other day, but then promptly forgot about it. I think the best thing at this point (hint hint!) would be for someone in the know to (re-)submit the various patches appropriate to move this forward. Whether that's just this series, part of it, or some/both of those + patches from you and Eric and this point I don't know/remember. But just to be explicitly clear, as probably the person most responsible for pushing this towards the "bigger question of [...] overall design". I just wanted to facilitate a discussion that would result in the various stakeholders who wanted to add some fsync-related config coming up with something that's mutually compatible, and I think the design from Neeraj in this series fits that purpose, is Good Enough etc. I.e. the actually important and IMO blockers were all resolved, e.g. not having an fsync configuration that older git versions would needlessly die on, and not painting ourselves into a corner where e.g. core.fsync=false or something was squatted on by something other than a "no fsync, whatsoever" etc. (But I haven't looked at it again just now, so...) Anyway, just trying to be explicit that to whatever extent this was held up by questions/comments of mine I'm very happy to see this go forward. As you (basically) say we shouldn't lose sight of ongoing data loss in this area because of some config bikeshedding :) ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v4 2/4] core.fsync: introduce granular fsync control 2022-03-09 13:42 ` Patrick Steinhardt 2022-03-09 18:50 ` Ævar Arnfjörð Bjarmason @ 2022-03-09 20:03 ` Junio C Hamano 2022-03-10 12:33 ` Patrick Steinhardt 2022-03-09 20:05 ` Neeraj Singh 2 siblings, 1 reply; 122+ messages in thread From: Junio C Hamano @ 2022-03-09 20:03 UTC (permalink / raw) To: Patrick Steinhardt Cc: Neeraj Singh, Neeraj Singh via GitGitGadget, Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Ævar Arnfjörð Bjarmason, Neeraj K. Singh Patrick Steinhardt <ps@pks.im> writes: >> If the user cares about fsynching loose object files in the right >> way, we shouldn't leave loose ref files not following the safe >> safety level, regardless of how this new core.fsync knobs would look >> like. >> >> I think we three are in agreement on that. > > Is there anything I can specifically do to help out with this topic? We > have again hit data loss in production because we don't sync loose refs > to disk before renaming them into place, so I'd really love to sort out > this issue somehow so that I can revive my patch series which fixes the > known repository corruption [1]. How about doing a series to unconditionally sync loose ref creation and modification? Alternatively, we could link it to the existing configuration to control synching of object files. I do not think core.fsyncObjectFiles having "object" in its name is a good reason not to think those who set it to true only care about the loose object files and nothing else. It is more sensible to consider that those who set it to true cares about the repository integrity more than those who set it to false, I would think. But that (i.e. doing it conditionally and choose which knob to use) is one extra thing that needs justification, so starting from unconditional fsync_or_die() may be the best way to ease it in. ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v4 2/4] core.fsync: introduce granular fsync control 2022-03-09 20:03 ` Junio C Hamano @ 2022-03-10 12:33 ` Patrick Steinhardt 2022-03-10 17:15 ` Junio C Hamano 0 siblings, 1 reply; 122+ messages in thread From: Patrick Steinhardt @ 2022-03-10 12:33 UTC (permalink / raw) To: Junio C Hamano Cc: Neeraj Singh, Neeraj Singh via GitGitGadget, Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Ævar Arnfjörð Bjarmason, Neeraj K. Singh [-- Attachment #1: Type: text/plain, Size: 1930 bytes --] On Wed, Mar 09, 2022 at 12:03:11PM -0800, Junio C Hamano wrote: > Patrick Steinhardt <ps@pks.im> writes: > > >> If the user cares about fsynching loose object files in the right > >> way, we shouldn't leave loose ref files not following the safe > >> safety level, regardless of how this new core.fsync knobs would look > >> like. > >> > >> I think we three are in agreement on that. > > > > Is there anything I can specifically do to help out with this topic? We > > have again hit data loss in production because we don't sync loose refs > > to disk before renaming them into place, so I'd really love to sort out > > this issue somehow so that I can revive my patch series which fixes the > > known repository corruption [1]. > > How about doing a series to unconditionally sync loose ref creation > and modification? > > Alternatively, we could link it to the existing configuration to > control synching of object files. > > I do not think core.fsyncObjectFiles having "object" in its name is > a good reason not to think those who set it to true only care about > the loose object files and nothing else. It is more sensible to > consider that those who set it to true cares about the repository > integrity more than those who set it to false, I would think. > > But that (i.e. doing it conditionally and choose which knob to use) > is one extra thing that needs justification, so starting from > unconditional fsync_or_die() may be the best way to ease it in. I'd be happy to revive my old patch series, but this kind of depends on where we're standing with the other patch series by Neeraj. If you say that we'll likely not land his patch series for the upcoming release, but a small patch series which only starts to sync loose refs may have a chance, then I'd like to go down that path as a stop-gap solution. Otherwise it probably wouldn't make a lot of sense. Patrick [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v4 2/4] core.fsync: introduce granular fsync control 2022-03-10 12:33 ` Patrick Steinhardt @ 2022-03-10 17:15 ` Junio C Hamano 0 siblings, 0 replies; 122+ messages in thread From: Junio C Hamano @ 2022-03-10 17:15 UTC (permalink / raw) To: Patrick Steinhardt Cc: Neeraj Singh, Neeraj Singh via GitGitGadget, Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Ævar Arnfjörð Bjarmason, Neeraj K. Singh Patrick Steinhardt <ps@pks.im> writes: >> How about doing a series to unconditionally sync loose ref creation >> and modification? >> >> Alternatively, we could link it to the existing configuration to >> control synching of object files. >> >> I do not think core.fsyncObjectFiles having "object" in its name is >> a good reason not to think those who set it to true only care about >> the loose object files and nothing else. It is more sensible to >> consider that those who set it to true cares about the repository >> integrity more than those who set it to false, I would think. >> >> But that (i.e. doing it conditionally and choose which knob to use) >> is one extra thing that needs justification, so starting from >> unconditional fsync_or_die() may be the best way to ease it in. > > I'd be happy to revive my old patch series, but this kind of depends on > where we're standing with the other patch series by Neeraj. If you say > that we'll likely not land his patch series for the upcoming release, > but a small patch series which only starts to sync loose refs may have a > chance, then I'd like to go down that path as a stop-gap solution. > Otherwise it probably wouldn't make a lot of sense. The above was what I wrote before the revived series from Neeraj. Now I've seen it, and more importantly, the most recent one from you on top of that to add ref hardening as a new "component" or two, I like the overall shape of the end result (except for the semantics implemented by the configuration parser, which can be fixed without affecting how the hardening components are implemented). Hopefully the base series of Neeraj can be solidified soon enough to make it unnecessary for a stop-gap measure. We'll see. Thanks. ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v4 2/4] core.fsync: introduce granular fsync control 2022-03-09 13:42 ` Patrick Steinhardt 2022-03-09 18:50 ` Ævar Arnfjörð Bjarmason 2022-03-09 20:03 ` Junio C Hamano @ 2022-03-09 20:05 ` Neeraj Singh 2 siblings, 0 replies; 122+ messages in thread From: Neeraj Singh @ 2022-03-09 20:05 UTC (permalink / raw) To: Patrick Steinhardt Cc: Junio C Hamano, Neeraj Singh via GitGitGadget, Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Ævar Arnfjörð Bjarmason, Neeraj K. Singh On Wed, Mar 9, 2022 at 5:42 AM Patrick Steinhardt <ps@pks.im> wrote: > > On Mon, Feb 14, 2022 at 09:17:31AM -0800, Junio C Hamano wrote: > > Patrick Steinhardt <ps@pks.im> writes: > > > > > To summarize my take: while the degree of durability may be something > > > that's up for discussions, I think that the current defaults for > > > atomicity are bad for users because they can and do lead to repository > > > corruption. > > > > Good summary. > > > > If the user cares about fsynching loose object files in the right > > way, we shouldn't leave loose ref files not following the safe > > safety level, regardless of how this new core.fsync knobs would look > > like. > > > > I think we three are in agreement on that. > > Is there anything I can specifically do to help out with this topic? We > have again hit data loss in production because we don't sync loose refs > to disk before renaming them into place, so I'd really love to sort out > this issue somehow so that I can revive my patch series which fixes the > known repository corruption [1]. > > Alternatively, can we maybe find a way forward with applying a version > of my patch series without first settling the bigger question of how we > want the overall design to look like? In my opinion repository > corruption is a severe bug that needs to be fixed, and it doesn't feel > sensible to block such a fix over a discussion that potentially will > take a long time to settle. > > Patrick > > [1]: http://public-inbox.org/git/cover.1636544377.git.ps@pks.im/ Hi Patrick, Thanks for reviving this discussion. I've updated the PR on GitGitGadget with a rebase onto the current 'main' branch and some minor build fixes. I've also revamped the aggregate options and documentation to be more inline with Junio's suggestion of having 'levels of safety' that we steer the user towards. I'm still keeping the detailed options, but hopefully the guidance is clear enough to avoid confusion. I'd be happy to make any point fixes as necessary to get that branch into proper shape for upstream, if we've gotten to the point where we don't want to change the fundamental design. I agree with Patrick that the detailed knobs are primarily for use by hosters like GitLab and GitHub. Please expect a v5 today. Thanks, Neeraj ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v4 2/4] core.fsync: introduce granular fsync control 2022-02-02 0:51 ` Junio C Hamano 2022-02-02 1:42 ` Junio C Hamano @ 2022-02-11 20:38 ` Neeraj Singh 1 sibling, 0 replies; 122+ messages in thread From: Neeraj Singh @ 2022-02-11 20:38 UTC (permalink / raw) To: Junio C Hamano Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Ævar Arnfjörð Bjarmason, Patrick Steinhardt, Neeraj K. Singh Apologies in advance for the delayed reply. I've finally been able to return to Git after an absence. On Tue, Feb 1, 2022 at 4:51 PM Junio C Hamano <gitster@pobox.com> wrote: > > "Neeraj Singh via GitGitGadget" <gitgitgadget@gmail.com> writes: > > > +core.fsync:: > > + A comma-separated list of parts of the repository which should be > > + hardened via the core.fsyncMethod when created or modified. You can > > + disable hardening of any component by prefixing it with a '-'. Later > > + items take precedence over earlier ones in the list. For example, > > + `core.fsync=all,-pack-metadata` means "harden everything except pack > > + metadata." Items that are not hardened may be lost in the event of an > > + unclean system shutdown. > > ++ > > +* `none` disables fsync completely. This must be specified alone. > > +* `loose-object` hardens objects added to the repo in loose-object form. > > +* `pack` hardens objects added to the repo in packfile form. > > +* `pack-metadata` hardens packfile bitmaps and indexes. > > +* `commit-graph` hardens the commit graph file. > > +* `objects` is an aggregate option that includes `loose-objects`, `pack`, > > + `pack-metadata`, and `commit-graph`. > > +* `default` is an aggregate option that is equivalent to `objects,-loose-object` > > +* `all` is an aggregate option that syncs all individual components above. > > I am not quite sure if this is way too complex (e.g. what does it > mean that we do not care much about loose-object safety while we do > care about commit-graph files?) and at the same time it is too > limited (e.g. if it makes sense to say a class of items deserve more > protection than another class of items, don't we want to be able to > say "class X is ultra-precious so use method A on them, while class > Y is mildly precious and use method B on them, everything else are > not that important and doing the default thing is just fine"). > > If we wanted to allow the "matrix" kind of flexibility, I think the > way to do so would be > > fsync.<class>.method = <value> > > e.g. > > [fsync "default"] method = none > [fsync "loose-object"] method = fsync > [fsync "pack-metadata"] method = writeout-only > I don't believe it makes sense to offer a full matrix of what to fsync and what method to use, since the method is a property of the filesystem and OS the repo is running on, while the list of things to fsync is more a selection of what the user values. So if I'm hosting on APFS on macOS or NTFS on Windows, I'd want to set the fsyncMethod to batch so that I can get good performance at the safety level I choose. If I'm working on my maintainer repo, I'd maybe not want to fsync anything, but I'd want to fsync everything when working on my developer repo. > Where do we expect users to take the core.fsync settings from? Per > repository? If it is from per user (i.e. $HOME/.gitconfig), do > people tend to share it across systems (not necessarily over NFS) > with the same contents? If so, I am not sure if fsync.method that > is way too close to the actual "implementation" is a good idea to > begin with. From end-user's point of view, it may be easier to > express "class X is ultra-precious, and class Y and Z are mildly > so", with something like fsync.<class>.level = <how-precious> and > let the Git implementation on each platform choose the appropriate > fsync method to protect the stuff at that precious-ness. > I expect the vast majority of users to have whatever setting is baked into their build of Git. For the users that want to do something different, I expect them to have core.fsyncMethod and core.fsync configured per-user for the majority of their repos. Some repos might have custom settings that override the per-user settings: 1) Ephemeral repos that don't contain unique data would probably want to set core.fsync=none. 2) Repos hosting on NFS or on a different FS may have a stricter core.fsyncmethod setting. (More more text to follow in reply to your next email). ^ permalink raw reply [flat|nested] 122+ messages in thread
* [PATCH v4 3/4] core.fsync: new option to harden the index 2022-02-01 3:33 ` [PATCH v4 " Neeraj K. Singh via GitGitGadget 2022-02-01 3:33 ` [PATCH v4 1/4] core.fsyncmethod: add writeout-only mode Neeraj Singh via GitGitGadget 2022-02-01 3:33 ` [PATCH v4 2/4] core.fsync: introduce granular fsync control Neeraj Singh via GitGitGadget @ 2022-02-01 3:33 ` Neeraj Singh via GitGitGadget 2022-02-01 3:33 ` [PATCH v4 4/4] core.fsync: add a `derived-metadata` aggregate option Neeraj Singh via GitGitGadget 2022-03-09 23:03 ` [PATCH v5 0/5] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget 4 siblings, 0 replies; 122+ messages in thread From: Neeraj Singh via GitGitGadget @ 2022-02-01 3:33 UTC (permalink / raw) To: git Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps, Neeraj K. Singh, Neeraj Singh From: Neeraj Singh <neerajsi@microsoft.com> This commit introduces the new ability for the user to harden the index. In the event of a system crash, the index must be durable for the user to actually find a file that has been added to the repo and then deleted from the working tree. We use the presence of the COMMIT_LOCK flag and absence of the alternate_index_output as a proxy for determining whether we're updating the persistent index of the repo or some temporary index. We don't sync these temporary indexes. Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> --- Documentation/config/core.txt | 1 + cache.h | 4 +++- config.c | 1 + read-cache.c | 19 +++++++++++++------ 4 files changed, 18 insertions(+), 7 deletions(-) diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt index 4f1747ec871..8e5b7a795ab 100644 --- a/Documentation/config/core.txt +++ b/Documentation/config/core.txt @@ -561,6 +561,7 @@ core.fsync:: * `pack` hardens objects added to the repo in packfile form. * `pack-metadata` hardens packfile bitmaps and indexes. * `commit-graph` hardens the commit graph file. +* `index` hardens the index when it is modified. * `objects` is an aggregate option that includes `loose-objects`, `pack`, `pack-metadata`, and `commit-graph`. * `default` is an aggregate option that is equivalent to `objects,-loose-object` diff --git a/cache.h b/cache.h index b3cd7d928de..9f3c1ec4c42 100644 --- a/cache.h +++ b/cache.h @@ -1004,6 +1004,7 @@ enum fsync_component { FSYNC_COMPONENT_PACK = 1 << 1, FSYNC_COMPONENT_PACK_METADATA = 1 << 2, FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3, + FSYNC_COMPONENT_INDEX = 1 << 4, }; #define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \ @@ -1018,7 +1019,8 @@ enum fsync_component { #define FSYNC_COMPONENTS_ALL (FSYNC_COMPONENT_LOOSE_OBJECT | \ FSYNC_COMPONENT_PACK | \ FSYNC_COMPONENT_PACK_METADATA | \ - FSYNC_COMPONENT_COMMIT_GRAPH) + FSYNC_COMPONENT_COMMIT_GRAPH | \ + FSYNC_COMPONENT_INDEX) /* diff --git a/config.c b/config.c index 224563c7b3e..325644e3c2c 100644 --- a/config.c +++ b/config.c @@ -1221,6 +1221,7 @@ static const struct fsync_component_entry { { "pack", FSYNC_COMPONENT_PACK }, { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA }, { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, + { "index", FSYNC_COMPONENT_INDEX }, { "objects", FSYNC_COMPONENTS_OBJECTS }, { "default", FSYNC_COMPONENTS_DEFAULT }, { "all", FSYNC_COMPONENTS_ALL }, diff --git a/read-cache.c b/read-cache.c index a0de70195c8..eb02439ab4b 100644 --- a/read-cache.c +++ b/read-cache.c @@ -2837,7 +2837,7 @@ static int record_ieot(void) * rely on it. */ static int do_write_index(struct index_state *istate, struct tempfile *tempfile, - int strip_extensions) + int strip_extensions, unsigned flags) { uint64_t start = getnanotime(); struct hashfile *f; @@ -2851,6 +2851,7 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile, struct strbuf previous_name_buf = STRBUF_INIT, *previous_name; int drop_cache_tree = istate->drop_cache_tree; off_t offset; + int csum_fsync_flag; int ieot_entries = 1; struct index_entry_offset_table *ieot = NULL; int nr, nr_threads; @@ -3081,7 +3082,13 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile, return -1; } - finalize_hashfile(f, istate->oid.hash, FSYNC_COMPONENT_NONE, CSUM_HASH_IN_STREAM); + csum_fsync_flag = 0; + if (!alternate_index_output && (flags & COMMIT_LOCK)) + csum_fsync_flag = CSUM_FSYNC; + + finalize_hashfile(f, istate->oid.hash, FSYNC_COMPONENT_INDEX, + CSUM_HASH_IN_STREAM | csum_fsync_flag); + if (close_tempfile_gently(tempfile)) { error(_("could not close '%s'"), get_tempfile_path(tempfile)); return -1; @@ -3136,7 +3143,7 @@ static int do_write_locked_index(struct index_state *istate, struct lock_file *l */ trace2_region_enter_printf("index", "do_write_index", the_repository, "%s", get_lock_file_path(lock)); - ret = do_write_index(istate, lock->tempfile, 0); + ret = do_write_index(istate, lock->tempfile, 0, flags); trace2_region_leave_printf("index", "do_write_index", the_repository, "%s", get_lock_file_path(lock)); @@ -3230,7 +3237,7 @@ static int clean_shared_index_files(const char *current_hex) } static int write_shared_index(struct index_state *istate, - struct tempfile **temp) + struct tempfile **temp, unsigned flags) { struct split_index *si = istate->split_index; int ret, was_full = !istate->sparse_index; @@ -3240,7 +3247,7 @@ static int write_shared_index(struct index_state *istate, trace2_region_enter_printf("index", "shared/do_write_index", the_repository, "%s", get_tempfile_path(*temp)); - ret = do_write_index(si->base, *temp, 1); + ret = do_write_index(si->base, *temp, 1, flags); trace2_region_leave_printf("index", "shared/do_write_index", the_repository, "%s", get_tempfile_path(*temp)); @@ -3349,7 +3356,7 @@ int write_locked_index(struct index_state *istate, struct lock_file *lock, ret = do_write_locked_index(istate, lock, flags); goto out; } - ret = write_shared_index(istate, &temp); + ret = write_shared_index(istate, &temp, flags); saved_errno = errno; if (is_tempfile_active(temp)) -- gitgitgadget ^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v4 4/4] core.fsync: add a `derived-metadata` aggregate option 2022-02-01 3:33 ` [PATCH v4 " Neeraj K. Singh via GitGitGadget ` (2 preceding siblings ...) 2022-02-01 3:33 ` [PATCH v4 3/4] core.fsync: new option to harden the index Neeraj Singh via GitGitGadget @ 2022-02-01 3:33 ` Neeraj Singh via GitGitGadget 2022-03-09 23:03 ` [PATCH v5 0/5] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget 4 siblings, 0 replies; 122+ messages in thread From: Neeraj Singh via GitGitGadget @ 2022-02-01 3:33 UTC (permalink / raw) To: git Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps, Neeraj K. Singh, Neeraj Singh From: Neeraj Singh <neerajsi@microsoft.com> This commit adds an aggregate option that currently includes the commit-graph file and pack metadata (indexes and bitmaps). The user may want to exclude this set from durability since they can be recomputed from other data if they wind up corrupt or missing. This is split out from the other patches in the series since it is an optional nice-to-have that might be controversial. Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> --- Documentation/config/core.txt | 6 +++--- cache.h | 7 ++++--- config.c | 1 + 3 files changed, 8 insertions(+), 6 deletions(-) diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt index 8e5b7a795ab..21092f3a4d1 100644 --- a/Documentation/config/core.txt +++ b/Documentation/config/core.txt @@ -562,9 +562,9 @@ core.fsync:: * `pack-metadata` hardens packfile bitmaps and indexes. * `commit-graph` hardens the commit graph file. * `index` hardens the index when it is modified. -* `objects` is an aggregate option that includes `loose-objects`, `pack`, - `pack-metadata`, and `commit-graph`. -* `default` is an aggregate option that is equivalent to `objects,-loose-object` +* `objects` is an aggregate option that includes `loose-objects` and `pack`. +* `derived-metadata` is an aggregate option that includes `pack-metadata` and `commit-graph`. +* `default` is an aggregate option that is equivalent to `objects,derived-metadata,-loose-object` * `all` is an aggregate option that syncs all individual components above. core.fsyncMethod:: diff --git a/cache.h b/cache.h index 9f3c1ec4c42..3327cf6af0b 100644 --- a/cache.h +++ b/cache.h @@ -1012,9 +1012,10 @@ enum fsync_component { FSYNC_COMPONENT_COMMIT_GRAPH) #define FSYNC_COMPONENTS_OBJECTS (FSYNC_COMPONENT_LOOSE_OBJECT | \ - FSYNC_COMPONENT_PACK | \ - FSYNC_COMPONENT_PACK_METADATA | \ - FSYNC_COMPONENT_COMMIT_GRAPH) + FSYNC_COMPONENT_PACK) + +#define FSYNC_COMPONENTS_DERIVED_METADATA (FSYNC_COMPONENT_PACK_METADATA | \ + FSYNC_COMPONENT_COMMIT_GRAPH) #define FSYNC_COMPONENTS_ALL (FSYNC_COMPONENT_LOOSE_OBJECT | \ FSYNC_COMPONENT_PACK | \ diff --git a/config.c b/config.c index 325644e3c2c..64a8a4d7c2a 100644 --- a/config.c +++ b/config.c @@ -1223,6 +1223,7 @@ static const struct fsync_component_entry { { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, { "index", FSYNC_COMPONENT_INDEX }, { "objects", FSYNC_COMPONENTS_OBJECTS }, + { "derived-metadata", FSYNC_COMPONENTS_DERIVED_METADATA }, { "default", FSYNC_COMPONENTS_DEFAULT }, { "all", FSYNC_COMPONENTS_ALL }, }; -- gitgitgadget ^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v5 0/5] A design for future-proofing fsync() configuration 2022-02-01 3:33 ` [PATCH v4 " Neeraj K. Singh via GitGitGadget ` (3 preceding siblings ...) 2022-02-01 3:33 ` [PATCH v4 4/4] core.fsync: add a `derived-metadata` aggregate option Neeraj Singh via GitGitGadget @ 2022-03-09 23:03 ` Neeraj K. Singh via GitGitGadget 2022-03-09 23:03 ` [PATCH v5 1/5] wrapper: move inclusion of CSPRNG headers the wrapper.c file Neeraj Singh via GitGitGadget ` (9 more replies) 4 siblings, 10 replies; 122+ messages in thread From: Neeraj K. Singh via GitGitGadget @ 2022-03-09 23:03 UTC (permalink / raw) To: git Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps, sandals, Neeraj K. Singh This is an implementation of an extensible configuration mechanism for fsyncing persistent components of a repo. The main goals are to separate the "what" to sync from the "how". There are now two settings: core.fsync - Control the 'what', including the index. core.fsyncMethod - Control the 'how'. Currently we support writeout-only and full fsync. Syncing of refs can be layered on top of core.fsync. And batch mode will be layered on core.fsyncMethod. Once this series reaches 'seen', I'll submit ns/batched-fsync to introduce batch mode. Please see https://github.com/gitgitgadget/git/pull/1134. core.fsyncObjectfiles is removed and we will issue a deprecation warning if it's seen. I'd like to get agreement on this direction before submitting batch mode to the list. The batch mode series is available to view at Please see [1], [2], and [3] for discussions that led to this series. After this change, new persistent data files added to the repo will need to be added to the fsync_component enum and documented in the Documentation/config/core.txt text. V5 changes: * Rebase onto main at c2162907e9 * Add a patch to move CSPRNG platform includes to wrapper.c. This avoids build errors in compat/win32/flush.c and other files. * Move the documentation and aggregate options to the final patch in the series. * Define new aggregate options and guidance in line with Junio's suggestion to present the user with 'levels of safety' rather than a morass of detailed options. V4 changes: * Rebase onto master at b23dac905bd. * Add a comment to write_pack_file indicating why we don't fsync when writing to stdout. * I kept the configuration schema as-is rather than switching to multi-value. The thinking here is that a stateless last-one-wins config schema (comma separated) will make it easier to achieve some holistic self-consistent fsync configuration for a particular repo. V3 changes: * Remove relative path from git-compat-util.h include [4]. * Updated newly added warning texts to have more context for localization [4]. * Fixed tab spacing in enum fsync_action * Moved the fsync looping out to a helper and do it consistently. [4] * Changed commit description to use camelCase for config names. [5] * Add an optional fourth patch with derived-metadata so that the user can exclude a forward-compatible set of things that should be recomputable given existing data. V2 changes: * Updated the documentation for core.fsyncmethod to be less certain. writeout-only probably does not do the right thing on Linux. * Split out the core.fsync=index change into its own commit. * Rename REPO_COMPONENT to FSYNC_COMPONENT. This is really specific to fsyncing, so the name should reflect that. * Re-add missing Makefile change for SYNC_FILE_RANGE. * Tested writeout-only mode, index syncing, and general config settings. [1] https://lore.kernel.org/git/211110.86r1bogg27.gmgdl@evledraar.gmail.com/ [2] https://lore.kernel.org/git/dd65718814011eb93ccc4428f9882e0f025224a6.1636029491.git.ps@pks.im/ [3] https://lore.kernel.org/git/pull.1076.git.git.1629856292.gitgitgadget@gmail.com/ [4] https://lore.kernel.org/git/CANQDOdf8C4-haK9=Q_J4Cid8bQALnmGDm=SvatRbaVf+tkzqLw@mail.gmail.com/ [5] https://lore.kernel.org/git/211207.861r2opplg.gmgdl@evledraar.gmail.com/ Neeraj Singh (5): wrapper: move inclusion of CSPRNG headers the wrapper.c file core.fsyncmethod: add writeout-only mode core.fsync: introduce granular fsync control core.fsync: new option to harden the index core.fsync: documentation and user-friendly aggregate options Documentation/config/core.txt | 51 +++++++++++++--- Makefile | 6 ++ builtin/fast-import.c | 2 +- builtin/index-pack.c | 4 +- builtin/pack-objects.c | 24 +++++--- bulk-checkin.c | 5 +- cache.h | 53 ++++++++++++++++- commit-graph.c | 3 +- compat/mingw.h | 3 + compat/win32/flush.c | 28 +++++++++ compat/winansi.c | 5 -- config.c | 92 ++++++++++++++++++++++++++++- config.mak.uname | 3 + configure.ac | 8 +++ contrib/buildsystems/CMakeLists.txt | 16 +++-- csum-file.c | 5 +- csum-file.h | 3 +- environment.c | 3 +- git-compat-util.h | 36 +++++++---- midx.c | 3 +- object-file.c | 3 +- pack-bitmap-write.c | 3 +- pack-write.c | 13 ++-- read-cache.c | 19 ++++-- wrapper.c | 78 ++++++++++++++++++++++++ write-or-die.c | 11 ++-- 26 files changed, 413 insertions(+), 67 deletions(-) create mode 100644 compat/win32/flush.c base-commit: c2162907e9aa884bdb70208389cb99b181620d51 Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1093%2Fneerajsi-msft%2Fns%2Fcore-fsync-v5 Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1093/neerajsi-msft/ns/core-fsync-v5 Pull-Request: https://github.com/gitgitgadget/git/pull/1093 Range-diff vs v4: -: ----------- > 1: 685b1db8880 wrapper: move inclusion of CSPRNG headers the wrapper.c file 1: 51a218d100d ! 2: da8cfc10bb4 core.fsyncmethod: add writeout-only mode @@ contrib/buildsystems/CMakeLists.txt @@ contrib/buildsystems/CMakeLists.txt: if(CMAKE_SYSTEM_NAME STREQUAL "Windows") NOGDI OBJECT_CREATION_MODE=1 __USE_MINGW_ANSI_STDIO=0 USE_NED_ALLOCATOR OVERRIDE_STRDUP MMAP_PREVENTS_DELETE USE_WIN32_MMAP - UNICODE _UNICODE HAVE_WPGMPTR ENSURE_MSYSTEM_IS_SET) + UNICODE _UNICODE HAVE_WPGMPTR ENSURE_MSYSTEM_IS_SET HAVE_RTLGENRANDOM) - list(APPEND compat_SOURCES compat/mingw.c compat/winansi.c compat/win32/path-utils.c -+ list(APPEND compat_SOURCES compat/mingw.c compat/winansi.c -+ compat/win32/flush.c compat/win32/path-utils.c - compat/win32/pthread.c compat/win32mmap.c compat/win32/syslog.c - compat/win32/trace2_win32_process_info.c compat/win32/dirent.c - compat/nedmalloc/nedmalloc.c compat/strdup.c) +- compat/win32/pthread.c compat/win32mmap.c compat/win32/syslog.c +- compat/win32/trace2_win32_process_info.c compat/win32/dirent.c +- compat/nedmalloc/nedmalloc.c compat/strdup.c) ++ list(APPEND compat_SOURCES ++ compat/mingw.c ++ compat/winansi.c ++ compat/win32/flush.c ++ compat/win32/path-utils.c ++ compat/win32/pthread.c ++ compat/win32mmap.c ++ compat/win32/syslog.c ++ compat/win32/trace2_win32_process_info.c ++ compat/win32/dirent.c ++ compat/nedmalloc/nedmalloc.c ++ compat/strdup.c) + set(NO_UNIX_SOCKETS 1) + + elseif(CMAKE_SYSTEM_NAME STREQUAL "Linux") ## environment.c ## @@ environment.c: int zlib_compression_level = Z_BEST_SPEED; @@ wrapper.c: int xmkstemp_mode(char *filename_template, int mode) + +#ifdef __APPLE__ + /* -+ * on macOS, fsync just causes filesystem cache writeback but does not -+ * flush hardware caches. ++ * On macOS, fsync just causes filesystem cache writeback but ++ * does not flush hardware caches. + */ + return fsync_loop(fd); +#endif + +#ifdef HAVE_SYNC_FILE_RANGE + /* -+ * On linux 2.6.17 and above, sync_file_range is the way to issue -+ * a writeback without a hardware flush. An offset of 0 and size of 0 -+ * indicates writeout of the entire file and the wait flags ensure that all -+ * dirty data is written to the disk (potentially in a disk-side cache) -+ * before we continue. ++ * On linux 2.6.17 and above, sync_file_range is the way to ++ * issue a writeback without a hardware flush. An offset of ++ * 0 and size of 0 indicates writeout of the entire file and the ++ * wait flags ensure that all dirty data is written to the disk ++ * (potentially in a disk-side cache) before we continue. + */ + + return sync_file_range(fd, 0, 0, SYNC_FILE_RANGE_WAIT_BEFORE | @@ wrapper.c: int xmkstemp_mode(char *filename_template, int mode) + + case FSYNC_HARDWARE_FLUSH: + /* -+ * On some platforms fsync may return EINTR. Try again in this -+ * case, since callers asking for a hardware flush may die if -+ * this function returns an error. ++ * On macOS, a special fcntl is required to really flush the ++ * caches within the storage controller. As of this writing, ++ * this is a very expensive operation on Apple SSDs. + */ +#ifdef __APPLE__ + return fcntl(fd, F_FULLFSYNC); 2: 7a164ba9571 ! 3: e31886717b4 core.fsync: introduce granular fsync control @@ Commit message syncable components: * We issue a warning rather than an error for unrecognized components, so new configs can be used with old Git versions. - * We support negation, so users can choose one of the default - aggregate options and then remove components that they don't - want. The user would then harden any new components added in - a Git version update. + * We support negation, so users can choose one of the aggregate + options and then remove components that they don't want. + Aggregate options are defined in a later patch in this series. This also supports the common request of doing absolutely no fysncing with the `core.fsync=none` value, which is expected to make the test suite faster. + Complete documentation for the new setting is included in a later patch + in the series so that it can be reviewed in final form. + Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> ## Documentation/config/core.txt ## -@@ Documentation/config/core.txt: core.whitespace:: - is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent` - errors. The default tab width is 8. Allowed values are 1 to 63. - -+core.fsync:: -+ A comma-separated list of parts of the repository which should be -+ hardened via the core.fsyncMethod when created or modified. You can -+ disable hardening of any component by prefixing it with a '-'. Later -+ items take precedence over earlier ones in the list. For example, -+ `core.fsync=all,-pack-metadata` means "harden everything except pack -+ metadata." Items that are not hardened may be lost in the event of an -+ unclean system shutdown. -++ -+* `none` disables fsync completely. This must be specified alone. -+* `loose-object` hardens objects added to the repo in loose-object form. -+* `pack` hardens objects added to the repo in packfile form. -+* `pack-metadata` hardens packfile bitmaps and indexes. -+* `commit-graph` hardens the commit graph file. -+* `objects` is an aggregate option that includes `loose-objects`, `pack`, -+ `pack-metadata`, and `commit-graph`. -+* `default` is an aggregate option that is equivalent to `objects,-loose-object` -+* `all` is an aggregate option that syncs all individual components above. -+ - core.fsyncMethod:: - A value indicating the strategy Git will use to harden repository data - using fsync and related primitives. @@ Documentation/config/core.txt: core.fsyncMethod:: filesystem and storage hardware, data added to the repository may not be durable in the event of a system crash. This is the default mode on macOS. @@ cache.h: void reset_shared_repository(void); + FSYNC_COMPONENT_PACK_METADATA | \ + FSYNC_COMPONENT_COMMIT_GRAPH) + -+#define FSYNC_COMPONENTS_OBJECTS (FSYNC_COMPONENT_LOOSE_OBJECT | \ -+ FSYNC_COMPONENT_PACK | \ -+ FSYNC_COMPONENT_PACK_METADATA | \ -+ FSYNC_COMPONENT_COMMIT_GRAPH) -+ -+#define FSYNC_COMPONENTS_ALL (FSYNC_COMPONENT_LOOSE_OBJECT | \ -+ FSYNC_COMPONENT_PACK | \ -+ FSYNC_COMPONENT_PACK_METADATA | \ -+ FSYNC_COMPONENT_COMMIT_GRAPH) -+ -+ +/* + * A bitmask indicating which components of the repo should be fsynced. + */ @@ cache.h: int copy_file_with_time(const char *dst, const char *src, int mode); void write_or_die(int fd, const void *buf, size_t count); void fsync_or_die(int fd, const char *); -+inline void fsync_component_or_die(enum fsync_component component, int fd, const char *msg) ++static inline void fsync_component_or_die(enum fsync_component component, int fd, const char *msg) +{ + if (fsync_components & component) + fsync_or_die(fd, msg); @@ config.c: static int git_parse_maybe_bool_text(const char *value) + { "pack", FSYNC_COMPONENT_PACK }, + { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA }, + { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, -+ { "objects", FSYNC_COMPONENTS_OBJECTS }, -+ { "default", FSYNC_COMPONENTS_DEFAULT }, -+ { "all", FSYNC_COMPONENTS_ALL }, +}; + +static enum fsync_component parse_fsync_components(const char *var, const char *string) @@ config.c: static int git_parse_maybe_bool_text(const char *value) + enum fsync_component output = 0; + + if (!strcmp(string, "none")) -+ return output; ++ return FSYNC_COMPONENT_NONE; + + while (string) { + int i; @@ midx.c: static int write_midx_internal(const char *object_dir, + CSUM_FSYNC | CSUM_HASH_IN_STREAM); free_chunkfile(cf); - if (flags & (MIDX_WRITE_REV_INDEX | MIDX_WRITE_BITMAP)) + if (flags & MIDX_WRITE_REV_INDEX && ## object-file.c ## @@ object-file.c: int hash_object_file(const struct git_hash_algo *algo, const void *buf, 3: f217dba77a1 ! 4: 9da808ba743 core.fsync: new option to harden the index @@ Commit message Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> - ## Documentation/config/core.txt ## -@@ Documentation/config/core.txt: core.fsync:: - * `pack` hardens objects added to the repo in packfile form. - * `pack-metadata` hardens packfile bitmaps and indexes. - * `commit-graph` hardens the commit graph file. -+* `index` hardens the index when it is modified. - * `objects` is an aggregate option that includes `loose-objects`, `pack`, - `pack-metadata`, and `commit-graph`. - * `default` is an aggregate option that is equivalent to `objects,-loose-object` - ## cache.h ## @@ cache.h: enum fsync_component { FSYNC_COMPONENT_PACK = 1 << 1, @@ cache.h: enum fsync_component { }; #define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \ -@@ cache.h: enum fsync_component { - #define FSYNC_COMPONENTS_ALL (FSYNC_COMPONENT_LOOSE_OBJECT | \ - FSYNC_COMPONENT_PACK | \ - FSYNC_COMPONENT_PACK_METADATA | \ -- FSYNC_COMPONENT_COMMIT_GRAPH) -+ FSYNC_COMPONENT_COMMIT_GRAPH | \ -+ FSYNC_COMPONENT_INDEX) - - - /* ## config.c ## @@ config.c: static const struct fsync_component_entry { @@ config.c: static const struct fsync_component_entry { { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA }, { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, + { "index", FSYNC_COMPONENT_INDEX }, - { "objects", FSYNC_COMPONENTS_OBJECTS }, - { "default", FSYNC_COMPONENTS_DEFAULT }, - { "all", FSYNC_COMPONENTS_ALL }, + }; + + static enum fsync_component parse_fsync_components(const char *var, const char *string) ## read-cache.c ## @@ read-cache.c: static int record_ieot(void) 4: 5c22a41c1f3 ! 5: 2d71346b10e core.fsync: add a `derived-metadata` aggregate option @@ Metadata Author: Neeraj Singh <neerajsi@microsoft.com> ## Commit message ## - core.fsync: add a `derived-metadata` aggregate option + core.fsync: documentation and user-friendly aggregate options - This commit adds an aggregate option that currently includes the - commit-graph file and pack metadata (indexes and bitmaps). + This commit adds aggregate options for the core.fsync setting that are + more user-friendly. These options are specified in terms of 'levels of + safety', indicating which Git operations are considered to be sync + points for durability. - The user may want to exclude this set from durability since they can be - recomputed from other data if they wind up corrupt or missing. - - This is split out from the other patches in the series since it is - an optional nice-to-have that might be controversial. + The new documentation is also included here in its entirety for ease of + review. Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> ## Documentation/config/core.txt ## -@@ Documentation/config/core.txt: core.fsync:: - * `pack-metadata` hardens packfile bitmaps and indexes. - * `commit-graph` hardens the commit graph file. - * `index` hardens the index when it is modified. --* `objects` is an aggregate option that includes `loose-objects`, `pack`, -- `pack-metadata`, and `commit-graph`. --* `default` is an aggregate option that is equivalent to `objects,-loose-object` -+* `objects` is an aggregate option that includes `loose-objects` and `pack`. -+* `derived-metadata` is an aggregate option that includes `pack-metadata` and `commit-graph`. -+* `default` is an aggregate option that is equivalent to `objects,derived-metadata,-loose-object` - * `all` is an aggregate option that syncs all individual components above. +@@ Documentation/config/core.txt: core.whitespace:: + is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent` + errors. The default tab width is 8. Allowed values are 1 to 63. ++core.fsync:: ++ A comma-separated list of parts of the repository which should be ++ hardened via the core.fsyncMethod when created or modified. You can ++ disable hardening of any component by prefixing it with a '-'. Later ++ items take precedence over earlier ones in the comma-separated list. ++ For example, `core.fsync=all,-pack-metadata` means "harden everything ++ except pack metadata." Items that are not hardened may be lost in the ++ event of an unclean system shutdown. Unless you have special ++ requirements, it is recommended that you leave this option as default ++ or pick one of `committed`, `added`, or `all`. +++ ++* `none` disables fsync completely. This value must be specified alone. ++* `loose-object` hardens objects added to the repo in loose-object form. ++* `pack` hardens objects added to the repo in packfile form. ++* `pack-metadata` hardens packfile bitmaps and indexes. ++* `commit-graph` hardens the commit graph file. ++* `index` hardens the index when it is modified. ++* `objects` is an aggregate option that is equivalent to ++ `loose-object,pack`. ++* `derived-metadata` is an aggregate option that is equivalent to ++ `pack-metadata,commit-graph`. ++* `default` is an aggregate option that is equivalent to ++ `objects,derived-metadata,-loose-object`. This mode is enabled by default. ++ It has good performance, but risks losing recent work if the system shuts ++ down uncleanly, since commits, trees, and blobs in loose-object form may be ++ lost. ++* `committed` is an aggregate option that is currently equivalent to ++ `objects`. This mode sacrifices some performance to ensure that all work ++ that is committed to the repository with `git commit` or similar commands ++ is preserved. ++* `added` is an aggregate option that is currently equivalent to ++ `committed,index`. This mode sacrifices additional performance to ++ ensure that the results of commands like `git add` and similar operations ++ are preserved. ++* `all` is an aggregate option that syncs all individual components above. ++ core.fsyncMethod:: + A value indicating the strategy Git will use to harden repository data + using fsync and related primitives. ## cache.h ## @@ cache.h: enum fsync_component { - FSYNC_COMPONENT_COMMIT_GRAPH) + FSYNC_COMPONENT_INDEX = 1 << 4, + }; - #define FSYNC_COMPONENTS_OBJECTS (FSYNC_COMPONENT_LOOSE_OBJECT | \ -- FSYNC_COMPONENT_PACK | \ +-#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \ - FSYNC_COMPONENT_PACK_METADATA | \ - FSYNC_COMPONENT_COMMIT_GRAPH) ++#define FSYNC_COMPONENTS_OBJECTS (FSYNC_COMPONENT_LOOSE_OBJECT | \ + FSYNC_COMPONENT_PACK) + +#define FSYNC_COMPONENTS_DERIVED_METADATA (FSYNC_COMPONENT_PACK_METADATA | \ + FSYNC_COMPONENT_COMMIT_GRAPH) ++ ++#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENTS_OBJECTS | \ ++ FSYNC_COMPONENTS_DERIVED_METADATA | \ ++ ~FSYNC_COMPONENT_LOOSE_OBJECT) ++ ++#define FSYNC_COMPONENTS_COMMITTED (FSYNC_COMPONENTS_OBJECTS) ++ ++#define FSYNC_COMPONENTS_ADDED (FSYNC_COMPONENTS_COMMITTED | \ ++ FSYNC_COMPONENT_INDEX) ++ ++#define FSYNC_COMPONENTS_ALL (FSYNC_COMPONENT_LOOSE_OBJECT | \ ++ FSYNC_COMPONENT_PACK | \ ++ FSYNC_COMPONENT_PACK_METADATA | \ ++ FSYNC_COMPONENT_COMMIT_GRAPH | \ ++ FSYNC_COMPONENT_INDEX) - #define FSYNC_COMPONENTS_ALL (FSYNC_COMPONENT_LOOSE_OBJECT | \ - FSYNC_COMPONENT_PACK | \ + /* + * A bitmask indicating which components of the repo should be fsynced. ## config.c ## @@ config.c: static const struct fsync_component_entry { + { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA }, { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, { "index", FSYNC_COMPONENT_INDEX }, - { "objects", FSYNC_COMPONENTS_OBJECTS }, ++ { "objects", FSYNC_COMPONENTS_OBJECTS }, + { "derived-metadata", FSYNC_COMPONENTS_DERIVED_METADATA }, - { "default", FSYNC_COMPONENTS_DEFAULT }, - { "all", FSYNC_COMPONENTS_ALL }, ++ { "default", FSYNC_COMPONENTS_DEFAULT }, ++ { "committed", FSYNC_COMPONENTS_COMMITTED }, ++ { "added", FSYNC_COMPONENTS_ADDED }, ++ { "all", FSYNC_COMPONENTS_ALL }, }; + + static enum fsync_component parse_fsync_components(const char *var, const char *string) -- gitgitgadget ^ permalink raw reply [flat|nested] 122+ messages in thread
* [PATCH v5 1/5] wrapper: move inclusion of CSPRNG headers the wrapper.c file 2022-03-09 23:03 ` [PATCH v5 0/5] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget @ 2022-03-09 23:03 ` Neeraj Singh via GitGitGadget 2022-03-09 23:29 ` Junio C Hamano 2022-03-10 1:26 ` brian m. carlson 2022-03-09 23:03 ` [PATCH v5 2/5] core.fsyncmethod: add writeout-only mode Neeraj Singh via GitGitGadget ` (8 subsequent siblings) 9 siblings, 2 replies; 122+ messages in thread From: Neeraj Singh via GitGitGadget @ 2022-03-09 23:03 UTC (permalink / raw) To: git Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps, sandals, Neeraj K. Singh, Neeraj Singh From: Neeraj Singh <neerajsi@microsoft.com> Including NTSecAPI.h in git-compat-util.h causes build errors in any other file that includes winternl.h. That file was included in order to get access to the RtlGenRandom cryptographically secure PRNG. This change scopes the inclusion of all PRNG headers to just the wrapper.c file, which is the only place it is really needed. Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> --- compat/winansi.c | 5 ----- git-compat-util.h | 12 ------------ wrapper.c | 14 ++++++++++++++ 3 files changed, 14 insertions(+), 17 deletions(-) diff --git a/compat/winansi.c b/compat/winansi.c index 936a80a5f00..3abe8dd5a27 100644 --- a/compat/winansi.c +++ b/compat/winansi.c @@ -4,11 +4,6 @@ #undef NOGDI -/* - * Including the appropriate header file for RtlGenRandom causes MSVC to see a - * redefinition of types in an incompatible way when including headers below. - */ -#undef HAVE_RTLGENRANDOM #include "../git-compat-util.h" #include <wingdi.h> #include <winreg.h> diff --git a/git-compat-util.h b/git-compat-util.h index 876907b9df4..a25ebb822ee 100644 --- a/git-compat-util.h +++ b/git-compat-util.h @@ -197,12 +197,6 @@ #endif #include <windows.h> #define GIT_WINDOWS_NATIVE -#ifdef HAVE_RTLGENRANDOM -/* This is required to get access to RtlGenRandom. */ -#define SystemFunction036 NTAPI SystemFunction036 -#include <NTSecAPI.h> -#undef SystemFunction036 -#endif #endif #include <unistd.h> @@ -273,12 +267,6 @@ #else #include <stdint.h> #endif -#ifdef HAVE_ARC4RANDOM_LIBBSD -#include <bsd/stdlib.h> -#endif -#ifdef HAVE_GETRANDOM -#include <sys/random.h> -#endif #ifdef NO_INTPTR_T /* * On I16LP32, ILP32 and LP64 "long" is the safe bet, however diff --git a/wrapper.c b/wrapper.c index 3258cdb171f..2a1aade473b 100644 --- a/wrapper.c +++ b/wrapper.c @@ -4,6 +4,20 @@ #include "cache.h" #include "config.h" +#ifdef HAVE_RTLGENRANDOM +/* This is required to get access to RtlGenRandom. */ +#define SystemFunction036 NTAPI SystemFunction036 +#include <NTSecAPI.h> +#undef SystemFunction036 +#endif + +#ifdef HAVE_ARC4RANDOM_LIBBSD +#include <bsd/stdlib.h> +#endif +#ifdef HAVE_GETRANDOM +#include <sys/random.h> +#endif + static int memory_limit_check(size_t size, int gentle) { static size_t limit = 0; -- gitgitgadget ^ permalink raw reply related [flat|nested] 122+ messages in thread
* Re: [PATCH v5 1/5] wrapper: move inclusion of CSPRNG headers the wrapper.c file 2022-03-09 23:03 ` [PATCH v5 1/5] wrapper: move inclusion of CSPRNG headers the wrapper.c file Neeraj Singh via GitGitGadget @ 2022-03-09 23:29 ` Junio C Hamano 2022-03-10 1:21 ` Neeraj Singh 2022-03-10 1:26 ` brian m. carlson 1 sibling, 1 reply; 122+ messages in thread From: Junio C Hamano @ 2022-03-09 23:29 UTC (permalink / raw) To: Neeraj Singh via GitGitGadget Cc: git, rsbecker, bagasdotme, newren, avarab, nksingh85, ps, sandals, Neeraj K. Singh "Neeraj Singh via GitGitGadget" <gitgitgadget@gmail.com> writes: > Including NTSecAPI.h in git-compat-util.h causes build errors in any > other file that includes winternl.h. That file was included in order to > get access to the RtlGenRandom cryptographically secure PRNG. This > change scopes the inclusion of all PRNG headers to just the wrapper.c > file, which is the only place it is really needed. It is true that wrapper.c is the only thing that needs these headers included as part of its implementation detail of csprng_bytes(), and I think I very much like this change for that reason. Having said that, if it true that including these two header files in the same file will lead to compilation failure? That sounds like either (1) they represent two mutually incompatible APIs that will cause breakage when code that use them, perhaps in two separate files to avoid compilation failures, are linked together, or (2) these system header files are simply broken. Or something else? > -/* > - * Including the appropriate header file for RtlGenRandom causes MSVC to see a > - * redefinition of types in an incompatible way when including headers below. > - */ > -#undef HAVE_RTLGENRANDOM This comment hints it is more like (1) above? A type used in one part of the system is defined differently in other parts of the system? I cannot imagine anything but bad things happen when a piece of code uses one definition of the type to declare a function, and another piece of code uses the other definition of the same type to declare a variable and passes it as a parameter to that function. I do not know this patch makes the situation worse, and I am not a Windows person with boxes to dig deeper to begin with. Hence I do not mind the change itself, but justifying the change primarily as a workaround for some obscure header type clashes on a single system leaves a bad taste. If the first sentence weren't there, I wouldn't have spent this many minutes wondering if this is a good change ;-) ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v5 1/5] wrapper: move inclusion of CSPRNG headers the wrapper.c file 2022-03-09 23:29 ` Junio C Hamano @ 2022-03-10 1:21 ` Neeraj Singh 0 siblings, 0 replies; 122+ messages in thread From: Neeraj Singh @ 2022-03-10 1:21 UTC (permalink / raw) To: Junio C Hamano Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Ævar Arnfjörð Bjarmason, Patrick Steinhardt, brian m. carlson, Neeraj K. Singh On Wed, Mar 9, 2022 at 3:29 PM Junio C Hamano <gitster@pobox.com> wrote: > > "Neeraj Singh via GitGitGadget" <gitgitgadget@gmail.com> writes: > > > Including NTSecAPI.h in git-compat-util.h causes build errors in any > > other file that includes winternl.h. That file was included in order to > > get access to the RtlGenRandom cryptographically secure PRNG. This > > change scopes the inclusion of all PRNG headers to just the wrapper.c > > file, which is the only place it is really needed. > > It is true that wrapper.c is the only thing that needs these headers > included as part of its implementation detail of csprng_bytes(), and > I think I very much like this change for that reason. > > Having said that, if it true that including these two header files > in the same file will lead to compilation failure? That sounds like > either (1) they represent two mutually incompatible APIs that will > cause breakage when code that use them, perhaps in two separate > files to avoid compilation failures, are linked together, or (2) > these system header files are simply broken. Or something else? > > > -/* > > - * Including the appropriate header file for RtlGenRandom causes MSVC to see a > > - * redefinition of types in an incompatible way when including headers below. > > - */ > > -#undef HAVE_RTLGENRANDOM > > This comment hints it is more like (1) above? A type used in one > part of the system is defined differently in other parts of the > system? I cannot imagine anything but bad things happen when a > piece of code uses one definition of the type to declare a function, > and another piece of code uses the other definition of the same type > to declare a variable and passes it as a parameter to that function. > > I do not know this patch makes the situation worse, and I am not a > Windows person with boxes to dig deeper to begin with. Hence I do > not mind the change itself, but justifying the change primarily as a > workaround for some obscure header type clashes on a single system > leaves a bad taste. If the first sentence weren't there, I wouldn't > have spent this many minutes wondering if this is a good change ;-) This is (2), these system header files are simply broken. I've been looking deeper into why, but haven't bottomed out yet. Thanks, Neeraj ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v5 1/5] wrapper: move inclusion of CSPRNG headers the wrapper.c file 2022-03-09 23:03 ` [PATCH v5 1/5] wrapper: move inclusion of CSPRNG headers the wrapper.c file Neeraj Singh via GitGitGadget 2022-03-09 23:29 ` Junio C Hamano @ 2022-03-10 1:26 ` brian m. carlson 2022-03-10 1:56 ` Neeraj Singh 1 sibling, 1 reply; 122+ messages in thread From: brian m. carlson @ 2022-03-10 1:26 UTC (permalink / raw) To: Neeraj Singh via GitGitGadget Cc: git, rsbecker, bagasdotme, newren, avarab, nksingh85, ps, Neeraj K. Singh [-- Attachment #1: Type: text/plain, Size: 1627 bytes --] On 2022-03-09 at 23:03:14, Neeraj Singh via GitGitGadget wrote: > From: Neeraj Singh <neerajsi@microsoft.com> > > Including NTSecAPI.h in git-compat-util.h causes build errors in any > other file that includes winternl.h. That file was included in order to > get access to the RtlGenRandom cryptographically secure PRNG. This > change scopes the inclusion of all PRNG headers to just the wrapper.c > file, which is the only place it is really needed. We generally prefer to do system includes in git-compat-util.h because it allows us to paper over platform incompatibilities in one place and to deal with the various ordering problems that can happen on certain systems. It may be that Windows needs additional help here; I don't know, because I don't use Windows. I personally find it unsavoury that Windows ships with multiple incompatible header files like this, since such problems are typically avoided by suitable include guards, whose utility has been well known for several decades. However, if that's the case, let's move only the Windows changes there, and leave the Unix systems, which lack this problem, alone. It would also be helpful to explain the problem that Windows has in more detail here, including any references to documentation that explains this incompatibility, so those of us who are not Windows users can more accurately reason about why we need to be so careful when including header files there and why this is the best solution (and not, say, providing our own include guards in a compat file). -- brian m. carlson (he/him or they/them) Toronto, Ontario, CA [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 262 bytes --] ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v5 1/5] wrapper: move inclusion of CSPRNG headers the wrapper.c file 2022-03-10 1:26 ` brian m. carlson @ 2022-03-10 1:56 ` Neeraj Singh 0 siblings, 0 replies; 122+ messages in thread From: Neeraj Singh @ 2022-03-10 1:56 UTC (permalink / raw) To: brian m. carlson, Neeraj Singh via GitGitGadget, Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Ævar Arnfjörð Bjarmason, Neeraj Singh, Patrick Steinhardt, Neeraj K. Singh On Wed, Mar 9, 2022 at 5:26 PM brian m. carlson <sandals@crustytoothpaste.net> wrote: > > On 2022-03-09 at 23:03:14, Neeraj Singh via GitGitGadget wrote: > > From: Neeraj Singh <neerajsi@microsoft.com> > > > > Including NTSecAPI.h in git-compat-util.h causes build errors in any > > other file that includes winternl.h. That file was included in order to > > get access to the RtlGenRandom cryptographically secure PRNG. This > > change scopes the inclusion of all PRNG headers to just the wrapper.c > > file, which is the only place it is really needed. > > We generally prefer to do system includes in git-compat-util.h because > it allows us to paper over platform incompatibilities in one place and > to deal with the various ordering problems that can happen on certain > systems. > > It may be that Windows needs additional help here; I don't know, because > I don't use Windows. I personally find it unsavoury that Windows ships > with multiple incompatible header files like this, since such problems > are typically avoided by suitable include guards, whose utility has been > well known for several decades. However, if that's the case, let's move > only the Windows changes there, and leave the Unix systems, which lack > this problem, alone. > > It would also be helpful to explain the problem that Windows has in more > detail here, including any references to documentation that explains > this incompatibility, so those of us who are not Windows users can more > accurately reason about why we need to be so careful when including > header files there and why this is the best solution (and not, say, > providing our own include guards in a compat file). > -- > brian m. carlson (he/him or they/them) > Toronto, Ontario, CA I wasn't able to find any documentation from other people who hit this problem. The root cause is that NtSecAPI.h has a typedef like this: ``` #ifndef _NTDEF_ typedef LSA_UNICODE_STRING UNICODE_STRING, *PUNICODE_STRING; typedef LSA_STRING STRING, *PSTRING ; #endif ``` That's not really appropriate since NtSecAPI.h isn't the correct place to define this core primitive NT type. It should be including winternl.h or a similar header. I'll update the change to only move the Windows definition to the .c file. ^ permalink raw reply [flat|nested] 122+ messages in thread
* [PATCH v5 2/5] core.fsyncmethod: add writeout-only mode 2022-03-09 23:03 ` [PATCH v5 0/5] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget 2022-03-09 23:03 ` [PATCH v5 1/5] wrapper: move inclusion of CSPRNG headers the wrapper.c file Neeraj Singh via GitGitGadget @ 2022-03-09 23:03 ` Neeraj Singh via GitGitGadget 2022-03-09 23:48 ` Junio C Hamano 2022-03-09 23:03 ` [PATCH v5 3/5] core.fsync: introduce granular fsync control Neeraj Singh via GitGitGadget ` (7 subsequent siblings) 9 siblings, 1 reply; 122+ messages in thread From: Neeraj Singh via GitGitGadget @ 2022-03-09 23:03 UTC (permalink / raw) To: git Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps, sandals, Neeraj K. Singh, Neeraj Singh From: Neeraj Singh <neerajsi@microsoft.com> This commit introduces the `core.fsyncMethod` configuration knob, which can currently be set to `fsync` or `writeout-only`. The new writeout-only mode attempts to tell the operating system to flush its in-memory page cache to the storage hardware without issuing a CACHE_FLUSH command to the storage controller. Writeout-only fsync is significantly faster than a vanilla fsync on common hardware, since data is written to a disk-side cache rather than all the way to a durable medium. Later changes in this patch series will take advantage of this primitive to implement batching of hardware flushes. When git_fsync is called with FSYNC_WRITEOUT_ONLY, it may fail and the caller is expected to do an ordinary fsync as needed. On Apple platforms, the fsync system call does not issue a CACHE_FLUSH directive to the storage controller. This change updates fsync to do fcntl(F_FULLFSYNC) to make fsync actually durable. We maintain parity with existing behavior on Apple platforms by setting the default value of the new core.fsyncMethod option. Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> --- Documentation/config/core.txt | 9 ++++ Makefile | 6 +++ cache.h | 7 ++++ compat/mingw.h | 3 ++ compat/win32/flush.c | 28 +++++++++++++ config.c | 12 ++++++ config.mak.uname | 3 ++ configure.ac | 8 ++++ contrib/buildsystems/CMakeLists.txt | 16 ++++++-- environment.c | 1 + git-compat-util.h | 24 +++++++++++ wrapper.c | 64 +++++++++++++++++++++++++++++ write-or-die.c | 11 +++-- 13 files changed, 184 insertions(+), 8 deletions(-) create mode 100644 compat/win32/flush.c diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt index c04f62a54a1..dbb134f7136 100644 --- a/Documentation/config/core.txt +++ b/Documentation/config/core.txt @@ -547,6 +547,15 @@ core.whitespace:: is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent` errors. The default tab width is 8. Allowed values are 1 to 63. +core.fsyncMethod:: + A value indicating the strategy Git will use to harden repository data + using fsync and related primitives. ++ +* `fsync` uses the fsync() system call or platform equivalents. +* `writeout-only` issues pagecache writeback requests, but depending on the + filesystem and storage hardware, data added to the repository may not be + durable in the event of a system crash. This is the default mode on macOS. + core.fsyncObjectFiles:: This boolean will enable 'fsync()' when writing object files. + diff --git a/Makefile b/Makefile index 6f0b4b775fe..17fd9b023a4 100644 --- a/Makefile +++ b/Makefile @@ -411,6 +411,8 @@ all:: # # Define HAVE_CLOCK_MONOTONIC if your platform has CLOCK_MONOTONIC. # +# Define HAVE_SYNC_FILE_RANGE if your platform has sync_file_range. +# # Define NEEDS_LIBRT if your platform requires linking with librt (glibc version # before 2.17) for clock_gettime and CLOCK_MONOTONIC. # @@ -1897,6 +1899,10 @@ ifdef HAVE_CLOCK_MONOTONIC BASIC_CFLAGS += -DHAVE_CLOCK_MONOTONIC endif +ifdef HAVE_SYNC_FILE_RANGE + BASIC_CFLAGS += -DHAVE_SYNC_FILE_RANGE +endif + ifdef NEEDS_LIBRT EXTLIBS += -lrt endif diff --git a/cache.h b/cache.h index 04d4d2db25c..82f0194a3dd 100644 --- a/cache.h +++ b/cache.h @@ -995,6 +995,13 @@ extern char *git_replace_ref_base; extern int fsync_object_files; extern int use_fsync; + +enum fsync_method { + FSYNC_METHOD_FSYNC, + FSYNC_METHOD_WRITEOUT_ONLY +}; + +extern enum fsync_method fsync_method; extern int core_preload_index; extern int precomposed_unicode; extern int protect_hfs; diff --git a/compat/mingw.h b/compat/mingw.h index c9a52ad64a6..6074a3d3ced 100644 --- a/compat/mingw.h +++ b/compat/mingw.h @@ -329,6 +329,9 @@ int mingw_getpagesize(void); #define getpagesize mingw_getpagesize #endif +int win32_fsync_no_flush(int fd); +#define fsync_no_flush win32_fsync_no_flush + struct rlimit { unsigned int rlim_cur; }; diff --git a/compat/win32/flush.c b/compat/win32/flush.c new file mode 100644 index 00000000000..291f90ea940 --- /dev/null +++ b/compat/win32/flush.c @@ -0,0 +1,28 @@ +#include "git-compat-util.h" +#include <winternl.h> +#include "lazyload.h" + +int win32_fsync_no_flush(int fd) +{ + IO_STATUS_BLOCK io_status; + +#define FLUSH_FLAGS_FILE_DATA_ONLY 1 + + DECLARE_PROC_ADDR(ntdll.dll, NTSTATUS, NTAPI, NtFlushBuffersFileEx, + HANDLE FileHandle, ULONG Flags, PVOID Parameters, ULONG ParameterSize, + PIO_STATUS_BLOCK IoStatusBlock); + + if (!INIT_PROC_ADDR(NtFlushBuffersFileEx)) { + errno = ENOSYS; + return -1; + } + + memset(&io_status, 0, sizeof(io_status)); + if (NtFlushBuffersFileEx((HANDLE)_get_osfhandle(fd), FLUSH_FLAGS_FILE_DATA_ONLY, + NULL, 0, &io_status)) { + errno = EINVAL; + return -1; + } + + return 0; +} diff --git a/config.c b/config.c index 383b1a4885b..f3ff80b01c9 100644 --- a/config.c +++ b/config.c @@ -1600,6 +1600,18 @@ static int git_default_core_config(const char *var, const char *value, void *cb) return 0; } + if (!strcmp(var, "core.fsyncmethod")) { + if (!value) + return config_error_nonbool(var); + if (!strcmp(value, "fsync")) + fsync_method = FSYNC_METHOD_FSYNC; + else if (!strcmp(value, "writeout-only")) + fsync_method = FSYNC_METHOD_WRITEOUT_ONLY; + else + warning(_("ignoring unknown core.fsyncMethod value '%s'"), value); + + } + if (!strcmp(var, "core.fsyncobjectfiles")) { fsync_object_files = git_config_bool(var, value); return 0; diff --git a/config.mak.uname b/config.mak.uname index 4352ea39e9b..404fff5dd04 100644 --- a/config.mak.uname +++ b/config.mak.uname @@ -57,6 +57,7 @@ ifeq ($(uname_S),Linux) HAVE_CLOCK_MONOTONIC = YesPlease # -lrt is needed for clock_gettime on glibc <= 2.16 NEEDS_LIBRT = YesPlease + HAVE_SYNC_FILE_RANGE = YesPlease HAVE_GETDELIM = YesPlease FREAD_READS_DIRECTORIES = UnfortunatelyYes BASIC_CFLAGS += -DHAVE_SYSINFO @@ -463,6 +464,7 @@ endif CFLAGS = BASIC_CFLAGS = -nologo -I. -Icompat/vcbuild/include -DWIN32 -D_CONSOLE -DHAVE_STRING_H -D_CRT_SECURE_NO_WARNINGS -D_CRT_NONSTDC_NO_DEPRECATE COMPAT_OBJS = compat/msvc.o compat/winansi.o \ + compat/win32/flush.o \ compat/win32/path-utils.o \ compat/win32/pthread.o compat/win32/syslog.o \ compat/win32/trace2_win32_process_info.o \ @@ -640,6 +642,7 @@ ifeq ($(uname_S),MINGW) COMPAT_CFLAGS += -DSTRIP_EXTENSION=\".exe\" COMPAT_OBJS += compat/mingw.o compat/winansi.o \ compat/win32/trace2_win32_process_info.o \ + compat/win32/flush.o \ compat/win32/path-utils.o \ compat/win32/pthread.o compat/win32/syslog.o \ compat/win32/dirent.o diff --git a/configure.ac b/configure.ac index 5ee25ec95c8..6bd6bef1c44 100644 --- a/configure.ac +++ b/configure.ac @@ -1082,6 +1082,14 @@ AC_COMPILE_IFELSE([CLOCK_MONOTONIC_SRC], [AC_MSG_RESULT([no]) HAVE_CLOCK_MONOTONIC=]) GIT_CONF_SUBST([HAVE_CLOCK_MONOTONIC]) + +# +# Define HAVE_SYNC_FILE_RANGE=YesPlease if sync_file_range is available. +GIT_CHECK_FUNC(sync_file_range, + [HAVE_SYNC_FILE_RANGE=YesPlease], + [HAVE_SYNC_FILE_RANGE]) +GIT_CONF_SUBST([HAVE_SYNC_FILE_RANGE]) + # # Define NO_SETITIMER if you don't have setitimer. GIT_CHECK_FUNC(setitimer, diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt index e44232f85d3..3a9e6241660 100644 --- a/contrib/buildsystems/CMakeLists.txt +++ b/contrib/buildsystems/CMakeLists.txt @@ -261,10 +261,18 @@ if(CMAKE_SYSTEM_NAME STREQUAL "Windows") NOGDI OBJECT_CREATION_MODE=1 __USE_MINGW_ANSI_STDIO=0 USE_NED_ALLOCATOR OVERRIDE_STRDUP MMAP_PREVENTS_DELETE USE_WIN32_MMAP UNICODE _UNICODE HAVE_WPGMPTR ENSURE_MSYSTEM_IS_SET HAVE_RTLGENRANDOM) - list(APPEND compat_SOURCES compat/mingw.c compat/winansi.c compat/win32/path-utils.c - compat/win32/pthread.c compat/win32mmap.c compat/win32/syslog.c - compat/win32/trace2_win32_process_info.c compat/win32/dirent.c - compat/nedmalloc/nedmalloc.c compat/strdup.c) + list(APPEND compat_SOURCES + compat/mingw.c + compat/winansi.c + compat/win32/flush.c + compat/win32/path-utils.c + compat/win32/pthread.c + compat/win32mmap.c + compat/win32/syslog.c + compat/win32/trace2_win32_process_info.c + compat/win32/dirent.c + compat/nedmalloc/nedmalloc.c + compat/strdup.c) set(NO_UNIX_SOCKETS 1) elseif(CMAKE_SYSTEM_NAME STREQUAL "Linux") diff --git a/environment.c b/environment.c index fd0501e77a5..3e3620d759f 100644 --- a/environment.c +++ b/environment.c @@ -44,6 +44,7 @@ int zlib_compression_level = Z_BEST_SPEED; int pack_compression_level = Z_DEFAULT_COMPRESSION; int fsync_object_files; int use_fsync = -1; +enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT; size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE; size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT; size_t delta_base_cache_limit = 96 * 1024 * 1024; diff --git a/git-compat-util.h b/git-compat-util.h index a25ebb822ee..41b97027457 100644 --- a/git-compat-util.h +++ b/git-compat-util.h @@ -1265,6 +1265,30 @@ __attribute__((format (printf, 1, 2))) NORETURN void BUG(const char *fmt, ...); #endif +#ifdef __APPLE__ +#define FSYNC_METHOD_DEFAULT FSYNC_METHOD_WRITEOUT_ONLY +#else +#define FSYNC_METHOD_DEFAULT FSYNC_METHOD_FSYNC +#endif + +enum fsync_action { + FSYNC_WRITEOUT_ONLY, + FSYNC_HARDWARE_FLUSH +}; + +/* + * Issues an fsync against the specified file according to the specified mode. + * + * FSYNC_WRITEOUT_ONLY attempts to use interfaces available on some operating + * systems to flush the OS cache without issuing a flush command to the storage + * controller. If those interfaces are unavailable, the function fails with + * ENOSYS. + * + * FSYNC_HARDWARE_FLUSH does an OS writeout and hardware flush to ensure that + * changes are durable. It is not expected to fail. + */ +int git_fsync(int fd, enum fsync_action action); + /* * Preserves errno, prints a message, but gives no warning for ENOENT. * Returns 0 on success, which includes trying to unlink an object that does diff --git a/wrapper.c b/wrapper.c index 2a1aade473b..2c44f71da42 100644 --- a/wrapper.c +++ b/wrapper.c @@ -553,6 +553,70 @@ int xmkstemp_mode(char *filename_template, int mode) return fd; } +/* + * Some platforms return EINTR from fsync. Since fsync is invoked in some + * cases by a wrapper that dies on failure, do not expose EINTR to callers. + */ +static int fsync_loop(int fd) +{ + int err; + + do { + err = fsync(fd); + } while (err < 0 && errno == EINTR); + return err; +} + +int git_fsync(int fd, enum fsync_action action) +{ + switch (action) { + case FSYNC_WRITEOUT_ONLY: + +#ifdef __APPLE__ + /* + * On macOS, fsync just causes filesystem cache writeback but + * does not flush hardware caches. + */ + return fsync_loop(fd); +#endif + +#ifdef HAVE_SYNC_FILE_RANGE + /* + * On linux 2.6.17 and above, sync_file_range is the way to + * issue a writeback without a hardware flush. An offset of + * 0 and size of 0 indicates writeout of the entire file and the + * wait flags ensure that all dirty data is written to the disk + * (potentially in a disk-side cache) before we continue. + */ + + return sync_file_range(fd, 0, 0, SYNC_FILE_RANGE_WAIT_BEFORE | + SYNC_FILE_RANGE_WRITE | + SYNC_FILE_RANGE_WAIT_AFTER); +#endif + +#ifdef fsync_no_flush + return fsync_no_flush(fd); +#endif + + errno = ENOSYS; + return -1; + + case FSYNC_HARDWARE_FLUSH: + /* + * On macOS, a special fcntl is required to really flush the + * caches within the storage controller. As of this writing, + * this is a very expensive operation on Apple SSDs. + */ +#ifdef __APPLE__ + return fcntl(fd, F_FULLFSYNC); +#else + return fsync_loop(fd); +#endif + default: + BUG("unexpected git_fsync(%d) call", action); + } +} + static int warn_if_unremovable(const char *op, const char *file, int rc) { int err; diff --git a/write-or-die.c b/write-or-die.c index a3d5784cec9..9faa5f9f563 100644 --- a/write-or-die.c +++ b/write-or-die.c @@ -62,10 +62,13 @@ void fsync_or_die(int fd, const char *msg) use_fsync = git_env_bool("GIT_TEST_FSYNC", 1); if (!use_fsync) return; - while (fsync(fd) < 0) { - if (errno != EINTR) - die_errno("fsync error on '%s'", msg); - } + + if (fsync_method == FSYNC_METHOD_WRITEOUT_ONLY && + git_fsync(fd, FSYNC_WRITEOUT_ONLY) >= 0) + return; + + if (git_fsync(fd, FSYNC_HARDWARE_FLUSH) < 0) + die_errno("fsync error on '%s'", msg); } void write_or_die(int fd, const void *buf, size_t count) -- gitgitgadget ^ permalink raw reply related [flat|nested] 122+ messages in thread
* Re: [PATCH v5 2/5] core.fsyncmethod: add writeout-only mode 2022-03-09 23:03 ` [PATCH v5 2/5] core.fsyncmethod: add writeout-only mode Neeraj Singh via GitGitGadget @ 2022-03-09 23:48 ` Junio C Hamano 0 siblings, 0 replies; 122+ messages in thread From: Junio C Hamano @ 2022-03-09 23:48 UTC (permalink / raw) To: Neeraj Singh via GitGitGadget Cc: git, rsbecker, bagasdotme, newren, avarab, nksingh85, ps, sandals, Neeraj K. Singh "Neeraj Singh via GitGitGadget" <gitgitgadget@gmail.com> writes: > From: Neeraj Singh <neerajsi@microsoft.com> > > This commit introduces the `core.fsyncMethod` configuration > knob, which can currently be set to `fsync` or `writeout-only`. > > The new writeout-only mode attempts to tell the operating system to > flush its in-memory page cache to the storage hardware without issuing a > CACHE_FLUSH command to the storage controller. > > Writeout-only fsync is significantly faster than a vanilla fsync on > common hardware, since data is written to a disk-side cache rather than > all the way to a durable medium. Later changes in this patch series will > take advantage of this primitive to implement batching of hardware > flushes. > > When git_fsync is called with FSYNC_WRITEOUT_ONLY, it may fail and the > caller is expected to do an ordinary fsync as needed. > > On Apple platforms, the fsync system call does not issue a CACHE_FLUSH > directive to the storage controller. This change updates fsync to do > fcntl(F_FULLFSYNC) to make fsync actually durable. We maintain parity > with existing behavior on Apple platforms by setting the default value > of the new core.fsyncMethod option. > > Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> > --- OK. This seems to be quite reasonable in that the pieces of code that use fsync_or_die() do not have to change at all, and all of them will keep behaving the same way. In other words, the "how" of fsync is very much well isolated. Nice. ^ permalink raw reply [flat|nested] 122+ messages in thread
* [PATCH v5 3/5] core.fsync: introduce granular fsync control 2022-03-09 23:03 ` [PATCH v5 0/5] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget 2022-03-09 23:03 ` [PATCH v5 1/5] wrapper: move inclusion of CSPRNG headers the wrapper.c file Neeraj Singh via GitGitGadget 2022-03-09 23:03 ` [PATCH v5 2/5] core.fsyncmethod: add writeout-only mode Neeraj Singh via GitGitGadget @ 2022-03-09 23:03 ` Neeraj Singh via GitGitGadget 2022-03-10 0:21 ` Junio C Hamano 2022-03-09 23:03 ` [PATCH v5 4/5] core.fsync: new option to harden the index Neeraj Singh via GitGitGadget ` (6 subsequent siblings) 9 siblings, 1 reply; 122+ messages in thread From: Neeraj Singh via GitGitGadget @ 2022-03-09 23:03 UTC (permalink / raw) To: git Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps, sandals, Neeraj K. Singh, Neeraj Singh From: Neeraj Singh <neerajsi@microsoft.com> This commit introduces the `core.fsync` configuration knob which can be used to control how components of the repository are made durable on disk. This setting allows future extensibility of the list of syncable components: * We issue a warning rather than an error for unrecognized components, so new configs can be used with old Git versions. * We support negation, so users can choose one of the aggregate options and then remove components that they don't want. Aggregate options are defined in a later patch in this series. This also supports the common request of doing absolutely no fysncing with the `core.fsync=none` value, which is expected to make the test suite faster. Complete documentation for the new setting is included in a later patch in the series so that it can be reviewed in final form. Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> --- Documentation/config/core.txt | 8 ---- builtin/fast-import.c | 2 +- builtin/index-pack.c | 4 +- builtin/pack-objects.c | 24 ++++++++---- bulk-checkin.c | 5 ++- cache.h | 30 +++++++++++++- commit-graph.c | 3 +- config.c | 73 ++++++++++++++++++++++++++++++++++- csum-file.c | 5 ++- csum-file.h | 3 +- environment.c | 2 +- midx.c | 3 +- object-file.c | 3 +- pack-bitmap-write.c | 3 +- pack-write.c | 13 ++++--- read-cache.c | 2 +- 16 files changed, 144 insertions(+), 39 deletions(-) diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt index dbb134f7136..74399072843 100644 --- a/Documentation/config/core.txt +++ b/Documentation/config/core.txt @@ -556,14 +556,6 @@ core.fsyncMethod:: filesystem and storage hardware, data added to the repository may not be durable in the event of a system crash. This is the default mode on macOS. -core.fsyncObjectFiles:: - This boolean will enable 'fsync()' when writing object files. -+ -This is a total waste of time and effort on a filesystem that orders -data writes properly, but can be useful for filesystems that do not use -journalling (traditional UNIX filesystems) or that only journal metadata -and not file contents (OS X's HFS+, or Linux ext3 with "data=writeback"). - core.preloadIndex:: Enable parallel index preload for operations like 'git diff' + diff --git a/builtin/fast-import.c b/builtin/fast-import.c index b7105fcad9b..f2c036a8955 100644 --- a/builtin/fast-import.c +++ b/builtin/fast-import.c @@ -865,7 +865,7 @@ static void end_packfile(void) struct tag *t; close_pack_windows(pack_data); - finalize_hashfile(pack_file, cur_pack_oid.hash, 0); + finalize_hashfile(pack_file, cur_pack_oid.hash, FSYNC_COMPONENT_PACK, 0); fixup_pack_header_footer(pack_data->pack_fd, pack_data->hash, pack_data->pack_name, object_count, cur_pack_oid.hash, pack_size); diff --git a/builtin/index-pack.c b/builtin/index-pack.c index c45273de3b1..c5f12f14df5 100644 --- a/builtin/index-pack.c +++ b/builtin/index-pack.c @@ -1290,7 +1290,7 @@ static void conclude_pack(int fix_thin_pack, const char *curr_pack, unsigned cha nr_objects - nr_objects_initial); stop_progress_msg(&progress, msg.buf); strbuf_release(&msg); - finalize_hashfile(f, tail_hash, 0); + finalize_hashfile(f, tail_hash, FSYNC_COMPONENT_PACK, 0); hashcpy(read_hash, pack_hash); fixup_pack_header_footer(output_fd, pack_hash, curr_pack, nr_objects, @@ -1512,7 +1512,7 @@ static void final(const char *final_pack_name, const char *curr_pack_name, if (!from_stdin) { close(input_fd); } else { - fsync_or_die(output_fd, curr_pack_name); + fsync_component_or_die(FSYNC_COMPONENT_PACK, output_fd, curr_pack_name); err = close(output_fd); if (err) die_errno(_("error while closing pack file")); diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 178e611f09d..c14fee8e99f 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -1199,16 +1199,26 @@ static void write_pack_file(void) display_progress(progress_state, written); } - /* - * Did we write the wrong # entries in the header? - * If so, rewrite it like in fast-import - */ if (pack_to_stdout) { - finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_CLOSE); + /* + * We never fsync when writing to stdout since we may + * not be writing to an actual pack file. For instance, + * the upload-pack code passes a pipe here. Calling + * fsync on a pipe results in unnecessary + * synchronization with the reader on some platforms. + */ + finalize_hashfile(f, hash, FSYNC_COMPONENT_NONE, + CSUM_HASH_IN_STREAM | CSUM_CLOSE); } else if (nr_written == nr_remaining) { - finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); + finalize_hashfile(f, hash, FSYNC_COMPONENT_PACK, + CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); } else { - int fd = finalize_hashfile(f, hash, 0); + /* + * If we wrote the wrong number of entries in the + * header, rewrite it like in fast-import. + */ + + int fd = finalize_hashfile(f, hash, FSYNC_COMPONENT_PACK, 0); fixup_pack_header_footer(fd, hash, pack_tmp_name, nr_written, hash, offset); close(fd); diff --git a/bulk-checkin.c b/bulk-checkin.c index 8785b2ac806..a2cf9dcbc8d 100644 --- a/bulk-checkin.c +++ b/bulk-checkin.c @@ -53,9 +53,10 @@ static void finish_bulk_checkin(struct bulk_checkin_state *state) unlink(state->pack_tmp_name); goto clear_exit; } else if (state->nr_written == 1) { - finalize_hashfile(state->f, hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); + finalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK, + CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); } else { - int fd = finalize_hashfile(state->f, hash, 0); + int fd = finalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK, 0); fixup_pack_header_footer(fd, hash, state->pack_tmp_name, state->nr_written, hash, state->offset); diff --git a/cache.h b/cache.h index 82f0194a3dd..a5eaa60a7a8 100644 --- a/cache.h +++ b/cache.h @@ -993,8 +993,27 @@ void reset_shared_repository(void); extern int read_replace_refs; extern char *git_replace_ref_base; -extern int fsync_object_files; -extern int use_fsync; +/* + * These values are used to help identify parts of a repository to fsync. + * FSYNC_COMPONENT_NONE identifies data that will not be a persistent part of the + * repository and so shouldn't be fsynced. + */ +enum fsync_component { + FSYNC_COMPONENT_NONE, + FSYNC_COMPONENT_LOOSE_OBJECT = 1 << 0, + FSYNC_COMPONENT_PACK = 1 << 1, + FSYNC_COMPONENT_PACK_METADATA = 1 << 2, + FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3, +}; + +#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \ + FSYNC_COMPONENT_PACK_METADATA | \ + FSYNC_COMPONENT_COMMIT_GRAPH) + +/* + * A bitmask indicating which components of the repo should be fsynced. + */ +extern enum fsync_component fsync_components; enum fsync_method { FSYNC_METHOD_FSYNC, @@ -1002,6 +1021,7 @@ enum fsync_method { }; extern enum fsync_method fsync_method; +extern int use_fsync; extern int core_preload_index; extern int precomposed_unicode; extern int protect_hfs; @@ -1708,6 +1728,12 @@ int copy_file_with_time(const char *dst, const char *src, int mode); void write_or_die(int fd, const void *buf, size_t count); void fsync_or_die(int fd, const char *); +static inline void fsync_component_or_die(enum fsync_component component, int fd, const char *msg) +{ + if (fsync_components & component) + fsync_or_die(fd, msg); +} + ssize_t read_in_full(int fd, void *buf, size_t count); ssize_t write_in_full(int fd, const void *buf, size_t count); ssize_t pread_in_full(int fd, void *buf, size_t count, off_t offset); diff --git a/commit-graph.c b/commit-graph.c index 265c010122e..64897f57d9f 100644 --- a/commit-graph.c +++ b/commit-graph.c @@ -1942,7 +1942,8 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx) } close_commit_graph(ctx->r->objects); - finalize_hashfile(f, file_hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC); + finalize_hashfile(f, file_hash, FSYNC_COMPONENT_COMMIT_GRAPH, + CSUM_HASH_IN_STREAM | CSUM_FSYNC); free_chunkfile(cf); if (ctx->split) { diff --git a/config.c b/config.c index f3ff80b01c9..51a35715642 100644 --- a/config.c +++ b/config.c @@ -1323,6 +1323,70 @@ static int git_parse_maybe_bool_text(const char *value) return -1; } +static const struct fsync_component_entry { + const char *name; + enum fsync_component component_bits; +} fsync_component_table[] = { + { "loose-object", FSYNC_COMPONENT_LOOSE_OBJECT }, + { "pack", FSYNC_COMPONENT_PACK }, + { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA }, + { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, +}; + +static enum fsync_component parse_fsync_components(const char *var, const char *string) +{ + enum fsync_component output = 0; + + if (!strcmp(string, "none")) + return FSYNC_COMPONENT_NONE; + + while (string) { + int i; + size_t len; + const char *ep; + int negated = 0; + int found = 0; + + string = string + strspn(string, ", \t\n\r"); + ep = strchrnul(string, ','); + len = ep - string; + + if (*string == '-') { + negated = 1; + string++; + len--; + if (!len) + warning(_("invalid value for variable %s"), var); + } + + if (!len) + break; + + for (i = 0; i < ARRAY_SIZE(fsync_component_table); ++i) { + const struct fsync_component_entry *entry = &fsync_component_table[i]; + + if (strncmp(entry->name, string, len)) + continue; + + found = 1; + if (negated) + output &= ~entry->component_bits; + else + output |= entry->component_bits; + } + + if (!found) { + char *component = xstrndup(string, len); + warning(_("ignoring unknown core.fsync component '%s'"), component); + free(component); + } + + string = ep; + } + + return output; +} + int git_parse_maybe_bool(const char *value) { int v = git_parse_maybe_bool_text(value); @@ -1600,6 +1664,13 @@ static int git_default_core_config(const char *var, const char *value, void *cb) return 0; } + if (!strcmp(var, "core.fsync")) { + if (!value) + return config_error_nonbool(var); + fsync_components = parse_fsync_components(var, value); + return 0; + } + if (!strcmp(var, "core.fsyncmethod")) { if (!value) return config_error_nonbool(var); @@ -1613,7 +1684,7 @@ static int git_default_core_config(const char *var, const char *value, void *cb) } if (!strcmp(var, "core.fsyncobjectfiles")) { - fsync_object_files = git_config_bool(var, value); + warning(_("core.fsyncobjectfiles is deprecated; use core.fsync instead")); return 0; } diff --git a/csum-file.c b/csum-file.c index 26e8a6df44e..59ef3398ca2 100644 --- a/csum-file.c +++ b/csum-file.c @@ -58,7 +58,8 @@ static void free_hashfile(struct hashfile *f) free(f); } -int finalize_hashfile(struct hashfile *f, unsigned char *result, unsigned int flags) +int finalize_hashfile(struct hashfile *f, unsigned char *result, + enum fsync_component component, unsigned int flags) { int fd; @@ -69,7 +70,7 @@ int finalize_hashfile(struct hashfile *f, unsigned char *result, unsigned int fl if (flags & CSUM_HASH_IN_STREAM) flush(f, f->buffer, the_hash_algo->rawsz); if (flags & CSUM_FSYNC) - fsync_or_die(f->fd, f->name); + fsync_component_or_die(component, f->fd, f->name); if (flags & CSUM_CLOSE) { if (close(f->fd)) die_errno("%s: sha1 file error on close", f->name); diff --git a/csum-file.h b/csum-file.h index 291215b34eb..0d29f528fbc 100644 --- a/csum-file.h +++ b/csum-file.h @@ -1,6 +1,7 @@ #ifndef CSUM_FILE_H #define CSUM_FILE_H +#include "cache.h" #include "hash.h" struct progress; @@ -38,7 +39,7 @@ int hashfile_truncate(struct hashfile *, struct hashfile_checkpoint *); struct hashfile *hashfd(int fd, const char *name); struct hashfile *hashfd_check(const char *name); struct hashfile *hashfd_throughput(int fd, const char *name, struct progress *tp); -int finalize_hashfile(struct hashfile *, unsigned char *, unsigned int); +int finalize_hashfile(struct hashfile *, unsigned char *, enum fsync_component, unsigned int); void hashwrite(struct hashfile *, const void *, unsigned int); void hashflush(struct hashfile *f); void crc32_begin(struct hashfile *); diff --git a/environment.c b/environment.c index 3e3620d759f..378424b9af5 100644 --- a/environment.c +++ b/environment.c @@ -42,9 +42,9 @@ const char *git_attributes_file; const char *git_hooks_path; int zlib_compression_level = Z_BEST_SPEED; int pack_compression_level = Z_DEFAULT_COMPRESSION; -int fsync_object_files; int use_fsync = -1; enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT; +enum fsync_component fsync_components = FSYNC_COMPONENTS_DEFAULT; size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE; size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT; size_t delta_base_cache_limit = 96 * 1024 * 1024; diff --git a/midx.c b/midx.c index 865170bad05..107365d2114 100644 --- a/midx.c +++ b/midx.c @@ -1438,7 +1438,8 @@ static int write_midx_internal(const char *object_dir, write_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs); write_chunkfile(cf, &ctx); - finalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM); + finalize_hashfile(f, midx_hash, FSYNC_COMPONENT_PACK_METADATA, + CSUM_FSYNC | CSUM_HASH_IN_STREAM); free_chunkfile(cf); if (flags & MIDX_WRITE_REV_INDEX && diff --git a/object-file.c b/object-file.c index 03bd6a3baf3..c9de912faca 100644 --- a/object-file.c +++ b/object-file.c @@ -1850,8 +1850,7 @@ int hash_object_file(const struct git_hash_algo *algo, const void *buf, static void close_loose_object(int fd) { if (!the_repository->objects->odb->will_destroy) { - if (fsync_object_files) - fsync_or_die(fd, "loose object file"); + fsync_component_or_die(FSYNC_COMPONENT_LOOSE_OBJECT, fd, "loose object file"); } if (close(fd) != 0) diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c index cab3eaa2acd..cf681547f2e 100644 --- a/pack-bitmap-write.c +++ b/pack-bitmap-write.c @@ -719,7 +719,8 @@ void bitmap_writer_finish(struct pack_idx_entry **index, if (options & BITMAP_OPT_HASH_CACHE) write_hash_cache(f, index, index_nr); - finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); + finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA, + CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); if (adjust_shared_perm(tmp_file.buf)) die_errno("unable to make temporary bitmap file readable"); diff --git a/pack-write.c b/pack-write.c index a5846f3a346..51812cb1299 100644 --- a/pack-write.c +++ b/pack-write.c @@ -159,9 +159,9 @@ const char *write_idx_file(const char *index_name, struct pack_idx_entry **objec } hashwrite(f, sha1, the_hash_algo->rawsz); - finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE | - ((opts->flags & WRITE_IDX_VERIFY) - ? 0 : CSUM_FSYNC)); + finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA, + CSUM_HASH_IN_STREAM | CSUM_CLOSE | + ((opts->flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC)); return index_name; } @@ -281,8 +281,9 @@ const char *write_rev_file_order(const char *rev_name, if (rev_name && adjust_shared_perm(rev_name) < 0) die(_("failed to make %s readable"), rev_name); - finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE | - ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC)); + finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA, + CSUM_HASH_IN_STREAM | CSUM_CLOSE | + ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC)); return rev_name; } @@ -390,7 +391,7 @@ void fixup_pack_header_footer(int pack_fd, the_hash_algo->final_fn(partial_pack_hash, &old_hash_ctx); the_hash_algo->final_fn(new_pack_hash, &new_hash_ctx); write_or_die(pack_fd, new_pack_hash, the_hash_algo->rawsz); - fsync_or_die(pack_fd, pack_name); + fsync_component_or_die(FSYNC_COMPONENT_PACK, pack_fd, pack_name); } char *index_pack_lockfile(int ip_out, int *is_well_formed) diff --git a/read-cache.c b/read-cache.c index 79b9b99ebf7..df869691fd4 100644 --- a/read-cache.c +++ b/read-cache.c @@ -3089,7 +3089,7 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile, return -1; } - finalize_hashfile(f, istate->oid.hash, CSUM_HASH_IN_STREAM); + finalize_hashfile(f, istate->oid.hash, FSYNC_COMPONENT_NONE, CSUM_HASH_IN_STREAM); if (close_tempfile_gently(tempfile)) { error(_("could not close '%s'"), get_tempfile_path(tempfile)); return -1; -- gitgitgadget ^ permalink raw reply related [flat|nested] 122+ messages in thread
* Re: [PATCH v5 3/5] core.fsync: introduce granular fsync control 2022-03-09 23:03 ` [PATCH v5 3/5] core.fsync: introduce granular fsync control Neeraj Singh via GitGitGadget @ 2022-03-10 0:21 ` Junio C Hamano 2022-03-10 2:53 ` Neeraj Singh 0 siblings, 1 reply; 122+ messages in thread From: Junio C Hamano @ 2022-03-10 0:21 UTC (permalink / raw) To: Neeraj Singh via GitGitGadget Cc: git, rsbecker, bagasdotme, newren, avarab, nksingh85, ps, sandals, Neeraj K. Singh "Neeraj Singh via GitGitGadget" <gitgitgadget@gmail.com> writes: > +/* > + * These values are used to help identify parts of a repository to fsync. > + * FSYNC_COMPONENT_NONE identifies data that will not be a persistent part of the > + * repository and so shouldn't be fsynced. > + */ > +enum fsync_component { > + FSYNC_COMPONENT_NONE, > + FSYNC_COMPONENT_LOOSE_OBJECT = 1 << 0, > + FSYNC_COMPONENT_PACK = 1 << 1, > + FSYNC_COMPONENT_PACK_METADATA = 1 << 2, > + FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3, > +}; OK, so the idea is that Patrick's "we need to fsync refs" will be done by adding a new component to this list, and sprinkling a call to fsync_component_or_die() in the code of ref-files backend? I am wondering if fsync_or_die() interface is abstracted well enough, or we need things like "the fd is inside this directory; in addition to doing the fsync of the fd, please sync the parent directory as well" support before we start adding more components (if there is such a need, perhaps it comes before this step). > +#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \ > + FSYNC_COMPONENT_PACK_METADATA | \ > + FSYNC_COMPONENT_COMMIT_GRAPH) IOW, everything other than loose object, which already has a separate core.fsyncObjectFiles knob to loosen. Everything else we currently sync unconditionally and the default keeps that arrangement? > +static inline void fsync_component_or_die(enum fsync_component component, int fd, const char *msg) > +{ > + if (fsync_components & component) > + fsync_or_die(fd, msg); > +} Do we have a compelling reason to have this as a static inline function? We are talking about concluding an I/O operation and I doubt there is a good performance argument for it. > +static const struct fsync_component_entry { > + const char *name; > + enum fsync_component component_bits; > +} fsync_component_table[] = { thing[] is an array of "thing" (and thing[4] is the "fourth" such thing), but this is not an array of a table (it is a name-to-bit mapping). I wonder if this array works without "_table" suffix in its name. > + { "loose-object", FSYNC_COMPONENT_LOOSE_OBJECT }, > + { "pack", FSYNC_COMPONENT_PACK }, > + { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA }, > + { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, > +}; > + > +static enum fsync_component parse_fsync_components(const char *var, const char *string) > +{ > + enum fsync_component output = 0; > + > + if (!strcmp(string, "none")) > + return FSYNC_COMPONENT_NONE; > + > + while (string) { > + int i; > + size_t len; > + const char *ep; > + int negated = 0; > + int found = 0; > + > + string = string + strspn(string, ", \t\n\r"); > + ep = strchrnul(string, ','); > + len = ep - string; > + > + if (*string == '-') { > + negated = 1; > + string++; > + len--; > + if (!len) > + warning(_("invalid value for variable %s"), var); > + } > + > + if (!len) > + break; > + > + for (i = 0; i < ARRAY_SIZE(fsync_component_table); ++i) { > + const struct fsync_component_entry *entry = &fsync_component_table[i]; > + > + if (strncmp(entry->name, string, len)) > + continue; > + > + found = 1; > + if (negated) > + output &= ~entry->component_bits; > + else > + output |= entry->component_bits; > + } > + > + if (!found) { > + char *component = xstrndup(string, len); > + warning(_("ignoring unknown core.fsync component '%s'"), component); > + free(component); > + } > + > + string = ep; > + } > + > + return output; > +} Hmph. I would have expected, with built-in default of pack,pack-metadata,commit-graph, - "none,pack" would choose only "pack" by first clearing the built-in default (or whatever was set in configuration files that are lower precedence than what we are reading) and then OR'ing the "pack" bit in. - "-pack" would choose "pack-metadata,commit-graph" by first starting from the built-in default and then CLR'ing the "pack" bit out. If there were already changes made by the lower precedence configuration files like /etc/gitconfig, the result might be different and the only definite thing we can say is that the pack bit is cleared. - "loose-object" would choose all of the bits by first starting from the built-in default and then OR'ing the "loose-object" bit in. Otherwise, parsing "none" is more or less pointless, as the above parser always start from 0 and OR's in or CLR's out the named bit. Whoever writes "none" can just write an empty string, no? I wonder you'd rather want to do it this way? parse_fsync_components(var, value, current) { enum fsync_component positive = 0, negative = 0; while (string) { int negated = 0; enum fsync_component bits; parse out a single component into <negated, bits>; if (bits == 0) { /* "none" given */ current = 0; } else if (negated) { negative |= bits; } else { positive |= bits; } advance <string> pointer; } return (current | positive) & ~negative; } And then ... > + if (!strcmp(var, "core.fsync")) { > + if (!value) > + return config_error_nonbool(var); > + fsync_components = parse_fsync_components(var, value); > + return 0; > + } > + ... this part would pass the current value of fsync_components as the third parameter to the parse_fsync_components(). The variable would be initialized to the FSYNC_COMPONENTS_DEFAULT we saw earlier. > @@ -1613,7 +1684,7 @@ static int git_default_core_config(const char *var, const char *value, void *cb) > } > > if (!strcmp(var, "core.fsyncobjectfiles")) { > - fsync_object_files = git_config_bool(var, value); > + warning(_("core.fsyncobjectfiles is deprecated; use core.fsync instead")); This is not deprecating but removing the support, which I am not sure is a sensible thing to do. Rather we should pretend that core.fsync = "loose-object" (or "-loose-object") were found in the configuration, shouldn't we? ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v5 3/5] core.fsync: introduce granular fsync control 2022-03-10 0:21 ` Junio C Hamano @ 2022-03-10 2:53 ` Neeraj Singh 2022-03-10 7:19 ` Junio C Hamano ` (2 more replies) 0 siblings, 3 replies; 122+ messages in thread From: Neeraj Singh @ 2022-03-10 2:53 UTC (permalink / raw) To: Junio C Hamano Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Ævar Arnfjörð Bjarmason, Patrick Steinhardt, brian m. carlson, Neeraj K. Singh On Wed, Mar 9, 2022 at 4:21 PM Junio C Hamano <gitster@pobox.com> wrote: > > "Neeraj Singh via GitGitGadget" <gitgitgadget@gmail.com> writes: > > > +/* > > + * These values are used to help identify parts of a repository to fsync. > > + * FSYNC_COMPONENT_NONE identifies data that will not be a persistent part of the > > + * repository and so shouldn't be fsynced. > > + */ > > +enum fsync_component { > > + FSYNC_COMPONENT_NONE, > > + FSYNC_COMPONENT_LOOSE_OBJECT = 1 << 0, > > + FSYNC_COMPONENT_PACK = 1 << 1, > > + FSYNC_COMPONENT_PACK_METADATA = 1 << 2, > > + FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3, > > +}; > > OK, so the idea is that Patrick's "we need to fsync refs" will be > done by adding a new component to this list, and sprinkling a call > to fsync_component_or_die() in the code of ref-files backend? > Yes. Patrick will need to add fsync_component_or_die wherever his patch series has already added fsync_or_die. If he follows Ævar's suggestion of treating remote refs differently from local refs, he might want to define multiple components. > I am wondering if fsync_or_die() interface is abstracted well > enough, or we need things like "the fd is inside this directory; in > addition to doing the fsync of the fd, please sync the parent > directory as well" support before we start adding more components > (if there is such a need, perhaps it comes before this step). > I think syncing the parent directory is a separate fsyncMethod that would require changes across the codebase to obtain an appropriate directory fd. I'd prefer to treat that as a separable concern. > > +#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \ > > + FSYNC_COMPONENT_PACK_METADATA | \ > > + FSYNC_COMPONENT_COMMIT_GRAPH) > > IOW, everything other than loose object, which already has a > separate core.fsyncObjectFiles knob to loosen. Everything else we > currently sync unconditionally and the default keeps that > arrangement? > Yes, trying to keep default behavior identical on non-Windows platforms. Windows will be expected to adopt batch mode, and have loose objects in this set. > > +static inline void fsync_component_or_die(enum fsync_component component, int fd, const char *msg) > > +{ > > + if (fsync_components & component) > > + fsync_or_die(fd, msg); > > +} > > Do we have a compelling reason to have this as a static inline > function? We are talking about concluding an I/O operation and > I doubt there is a good performance argument for it. > Yeah, this is meant to optimize the case where the component isn't being fsynced. I'll move this function to write-or-die.c below fsync_or_die. > > +static const struct fsync_component_entry { > > + const char *name; > > + enum fsync_component component_bits; > > +} fsync_component_table[] = { > > thing[] is an array of "thing" (and thing[4] is the "fourth" such > thing), but this is not an array of a table (it is a name-to-bit > mapping). > > I wonder if this array works without "_table" suffix in its name. This is modeled after whitespace_rule_names. What if I change this to the following? static const struct fsync_component_name { ... } fsync_component_names[] > > > + { "loose-object", FSYNC_COMPONENT_LOOSE_OBJECT }, > > + { "pack", FSYNC_COMPONENT_PACK }, > > + { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA }, > > + { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, > > +}; > > + > > +static enum fsync_component parse_fsync_components(const char *var, const char *string) > > +{ > > + enum fsync_component output = 0; > > + > > + if (!strcmp(string, "none")) > > + return FSYNC_COMPONENT_NONE; > > + > > + while (string) { > > + int i; > > + size_t len; > > + const char *ep; > > + int negated = 0; > > + int found = 0; > > + > > + string = string + strspn(string, ", \t\n\r"); > > + ep = strchrnul(string, ','); > > + len = ep - string; > > + > > + if (*string == '-') { > > + negated = 1; > > + string++; > > + len--; > > + if (!len) > > + warning(_("invalid value for variable %s"), var); > > + } > > + > > + if (!len) > > + break; > > + > > + for (i = 0; i < ARRAY_SIZE(fsync_component_table); ++i) { > > + const struct fsync_component_entry *entry = &fsync_component_table[i]; > > + > > + if (strncmp(entry->name, string, len)) > > + continue; > > + > > + found = 1; > > + if (negated) > > + output &= ~entry->component_bits; > > + else > > + output |= entry->component_bits; > > + } > > + > > + if (!found) { > > + char *component = xstrndup(string, len); > > + warning(_("ignoring unknown core.fsync component '%s'"), component); > > + free(component); > > + } > > + > > + string = ep; > > + } > > + > > + return output; > > +} > > Hmph. I would have expected, with built-in default of > pack,pack-metadata,commit-graph, > At the conclusion of this series, I defined 'default' as an aggregate option that includes the platform default. I'd prefer not to have any statefulness of the core.fsync setting so that there is less confusion about the final fsync configuration. My colleagues had a fair amount of confusion internally when testing Git performance internally with regards to the core.fsyncObjectFiles setting. Inline this is how your configs would be written: > - "none,pack" would choose only "pack" by first clearing the > built-in default (or whatever was set in configuration files that > are lower precedence than what we are reading) and then OR'ing > the "pack" bit in. > "pack" would choose only "pack" > - "-pack" would choose "pack-metadata,commit-graph" by first > starting from the built-in default and then CLR'ing the "pack" > bit out. If there were already changes made by the lower > precedence configuration files like /etc/gitconfig, the result > might be different and the only definite thing we can say is that > the pack bit is cleared. > "default,-pack" would be the platform default, but not packfiles. > - "loose-object" would choose all of the bits by first starting > from the built-in default and then OR'ing the "loose-object" bit > in. > "default,loose-object" would add loose objects to the platform config. > Otherwise, parsing "none" is more or less pointless, as the above > parser always start from 0 and OR's in or CLR's out the named bit. > Whoever writes "none" can just write an empty string, no? I think the empty string would be disallowed since "core.fsync=" would be entirely missing a value. But on testing this doesn't seem to be the case. I'll change this to be more strict in that the user has to pass an explicit value, such as 'none'. > > I wonder you'd rather want to do it this way? > > parse_fsync_components(var, value, current) { > enum fsync_component positive = 0, negative = 0; > > while (string) { > int negated = 0; > enum fsync_component bits; > > parse out a single component into <negated, bits>; > > if (bits == 0) { /* "none" given */ > current = 0; > } else if (negated) { > negative |= bits; > } else { > positive |= bits; > } > advance <string> pointer; > } > > return (current | positive) & ~negative; > } > > And then ... > > > + if (!strcmp(var, "core.fsync")) { > > + if (!value) > > + return config_error_nonbool(var); > > + fsync_components = parse_fsync_components(var, value); > > + return 0; > > + } > > + > > ... this part would pass the current value of fsync_components as > the third parameter to the parse_fsync_components(). The variable > would be initialized to the FSYNC_COMPONENTS_DEFAULT we saw earlier. > I'm afraid that this method would lead to statefulness between different levels configuring core.fsync. I'd prefer that the user could know what will happen by just inspecting the value returned by `git config core.fsync`. > > > @@ -1613,7 +1684,7 @@ static int git_default_core_config(const char *var, const char *value, void *cb) > > } > > > > if (!strcmp(var, "core.fsyncobjectfiles")) { > > - fsync_object_files = git_config_bool(var, value); > > + warning(_("core.fsyncobjectfiles is deprecated; use core.fsync instead")); > > This is not deprecating but removing the support, which I am not > sure is a sensible thing to do. Rather we should pretend that > core.fsync = "loose-object" (or "-loose-object") were found in the > configuration, shouldn't we? > The problem I anticipate is that figuring out the final configuration becomes pretty complex in the face of conflicting configurations of fsyncObjectFiles and core.fsync. The user won't know what will happen without reading the Git code or doing performance experiments. I thought we can avoid all of this complexity by just having a simple warning that pushes users toward the new configuration value. Aside from seeing a warning, a user's actual usage of Git functionality shouldn't be affected. Alternatively, what if we just silently ignore the old core.fsyncObjectFiles setting? If neither of these is an option, I'll put back some support for the old setting. Thanks, Neeraj ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v5 3/5] core.fsync: introduce granular fsync control 2022-03-10 2:53 ` Neeraj Singh @ 2022-03-10 7:19 ` Junio C Hamano 2022-03-10 18:38 ` Neeraj Singh 2022-03-10 13:11 ` Johannes Schindelin 2022-03-10 17:18 ` Junio C Hamano 2 siblings, 1 reply; 122+ messages in thread From: Junio C Hamano @ 2022-03-10 7:19 UTC (permalink / raw) To: Neeraj Singh Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Ævar Arnfjörð Bjarmason, Patrick Steinhardt, brian m. carlson, Neeraj K. Singh Neeraj Singh <nksingh85@gmail.com> writes: > This is modeled after whitespace_rule_names. What if I change this to > the following? > static const struct fsync_component_name { > ... > } fsync_component_names[] That's better. > At the conclusion of this series, I defined 'default' as an aggregate > option that includes > the platform default. I'd prefer not to have any statefulness of the > core.fsync setting so > that there is less confusion about the final fsync configuration. Then scratch your preference ;-) Our configuration files are designed to be "hierarchical" in that system-wide default can be set in /etc/gitconfig, which can be overridden by per-user default in $HOME/.gitconfig, which can in turn be overridden by per-repository setting in .git/config, so starting from the compiled-in default, reading/augmenting "the value we tentatively decided based on what we have read so far" with what we read from lower-precedence files to higher-precedence files is a norm. Don't make this little corner of the system different from everything else; that will only confuse users. The git_config() callback should expect to see the same var with different values for that reason. Always restarting from zero will defeat it. And always restarting from zero will mean "none" is meaningless, while it would be a quite nice way to say "forget everything we have read so far and start from scratch" when you really want to refuse what the system default wants to give you. >> > @@ -1613,7 +1684,7 @@ static int git_default_core_config(const char *var, const char *value, void *cb) >> > } >> > >> > if (!strcmp(var, "core.fsyncobjectfiles")) { >> > - fsync_object_files = git_config_bool(var, value); >> > + warning(_("core.fsyncobjectfiles is deprecated; use core.fsync instead")); >> >> This is not deprecating but removing the support, which I am not >> sure is a sensible thing to do. Rather we should pretend that >> core.fsync = "loose-object" (or "-loose-object") were found in the >> configuration, shouldn't we? >> > > The problem I anticipate is that figuring out the final configuration > becomes pretty > complex in the face of conflicting configurations of fsyncObjectFiles > and core.fsync. Don't start your thinking from too complex configuration that mixes and matches. Instead, think what happens to folks who are *NOT* interested in the new way of doing this. They aren't interested in setting core.fsync, and they already have core.fsyncObjectFiles set. You want to make sure their experience does not suck. The quoted code simply _IGNORES_ their wish and forces whatever default configuration core.fsync codepath happens to choose, which is a grave regression from their point of view. One way to handle this more gracefully is to delay the final decision until the end of the configuraiton file processing, and keep track of core.fsyncObjectFiles and core.fsync separately. If the latter is never set but the former is, then you are dealing with such a user who hasn't migrated. Give them a warning (the text above is fine---we can tell them "that's deprecated and you should use the other one instead"), but in the meantime, until deprecation turns into removal of support, keep honoring their original wish. If core.fsync is set to something, you can still give them a warning when you see core.fsyncObjectFiles, saying "that's deprecated and because you have core.fsync, we'll ignore the old one", and use the new method exclusively, without having to worry about mixing. ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v5 3/5] core.fsync: introduce granular fsync control 2022-03-10 7:19 ` Junio C Hamano @ 2022-03-10 18:38 ` Neeraj Singh 2022-03-10 18:44 ` Junio C Hamano 0 siblings, 1 reply; 122+ messages in thread From: Neeraj Singh @ 2022-03-10 18:38 UTC (permalink / raw) To: Junio C Hamano Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Ævar Arnfjörð Bjarmason, Patrick Steinhardt, brian m. carlson, Neeraj K. Singh On Wed, Mar 9, 2022 at 11:19 PM Junio C Hamano <gitster@pobox.com> wrote: > > Neeraj Singh <nksingh85@gmail.com> writes: > > > This is modeled after whitespace_rule_names. What if I change this to > > the following? > > static const struct fsync_component_name { > > ... > > } fsync_component_names[] > > That's better. > > > At the conclusion of this series, I defined 'default' as an aggregate > > option that includes > > the platform default. I'd prefer not to have any statefulness of the > > core.fsync setting so > > that there is less confusion about the final fsync configuration. > > Then scratch your preference ;-) Just to clarify, linguistically, by 'scratch' do you mean that I should drop my preference (which I can do to get the important parts of the series in), or are you saying that that your suggestion is in line with my preference, and I'm just not seeing it properly? > Our configuration files are designed to be "hierarchical" in that > system-wide default can be set in /etc/gitconfig, which can be > overridden by per-user default in $HOME/.gitconfig, which can in > turn be overridden by per-repository setting in .git/config, so > starting from the compiled-in default, reading/augmenting "the value > we tentatively decided based on what we have read so far" with what > we read from lower-precedence files to higher-precedence files is a > norm. > > Don't make this little corner of the system different from > everything else; that will only confuse users. > > The git_config() callback should expect to see the same var with > different values for that reason. Always restarting from zero will > defeat it. > Consider core.whitespace. The parse_whitespace_rule code starts with the compiled-in default every time it encounters a config value, and then modifies it according to what the user passed in. So the user could figure out what's going to happen by just looking at the value returned by `git config --get core.whitespace`. The user doesn't need to call `git config --get-all core.whitespace` and then reason about the entries from top to bottom to figure out the actual state that Git will use. > And always restarting from zero will mean "none" is meaningless, > while it would be a quite nice way to say "forget everything we have > read so far and start from scratch" when you really want to refuse > what the system default wants to give you. > The intention, which I've implemented in my local v6 changes, is for an empty list or empty string to be an illegal value of core.fsync. It should be set to one or more legal values. But I see the advantage in always resetting to the system default, like core.whitespace does, so that a set of unrecognized values results in at least default behavior. An empty string would mean 'unconfigured', which will be meaningful when we integrate core.fsync with core.fsyncObjectFiles. I'll update the change to start from default, with none as a reset. I'm still in favor of making it so that the most recent value of core.fsync in the hierarchical configuration store stands alone without picking up state from prior values. > >> > @@ -1613,7 +1684,7 @@ static int git_default_core_config(const char *var, const char *value, void *cb) > >> > } > >> > > >> > if (!strcmp(var, "core.fsyncobjectfiles")) { > >> > - fsync_object_files = git_config_bool(var, value); > >> > + warning(_("core.fsyncobjectfiles is deprecated; use core.fsync instead")); > >> > >> This is not deprecating but removing the support, which I am not > >> sure is a sensible thing to do. Rather we should pretend that > >> core.fsync = "loose-object" (or "-loose-object") were found in the > >> configuration, shouldn't we? > >> > > > > The problem I anticipate is that figuring out the final configuration > > becomes pretty > > complex in the face of conflicting configurations of fsyncObjectFiles > > and core.fsync. > > Don't start your thinking from too complex configuration that mixes > and matches. Instead, think what happens to folks who are *NOT* > interested in the new way of doing this. They aren't interested in > setting core.fsync, and they already have core.fsyncObjectFiles set. > You want to make sure their experience does not suck. > > The quoted code simply _IGNORES_ their wish and forces whatever > default configuration core.fsync codepath happens to choose, which > is a grave regression from their point of view. > > One way to handle this more gracefully is to delay the final > decision until the end of the configuraiton file processing, and > keep track of core.fsyncObjectFiles and core.fsync separately. If > the latter is never set but the former is, then you are dealing with > such a user who hasn't migrated. Give them a warning (the text > above is fine---we can tell them "that's deprecated and you should > use the other one instead"), but in the meantime, until deprecation > turns into removal of support, keep honoring their original wish. > If core.fsync is set to something, you can still give them a warning > when you see core.fsyncObjectFiles, saying "that's deprecated and > because you have core.fsync, we'll ignore the old one", and use the > new method exclusively, without having to worry about mixing. Is there a well-defined place where we know that configuration processing is complete? The most obvious spot to me to integrate these two values would be the first time we need to figure out the fsync state. Another spot would be prepare_repo_settings. Are there any other good candidates? Once the right spot is picked, I'll implement the integration of the settings as you suggested. For now I'll stick it in prepare_repo_settings. Thanks for the review. Please expect a v6 today. ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v5 3/5] core.fsync: introduce granular fsync control 2022-03-10 18:38 ` Neeraj Singh @ 2022-03-10 18:44 ` Junio C Hamano 2022-03-10 19:57 ` Junio C Hamano 0 siblings, 1 reply; 122+ messages in thread From: Junio C Hamano @ 2022-03-10 18:44 UTC (permalink / raw) To: Neeraj Singh Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Ævar Arnfjörð Bjarmason, Patrick Steinhardt, brian m. carlson, Neeraj K. Singh Neeraj Singh <nksingh85@gmail.com> writes: >> > At the conclusion of this series, I defined 'default' as an aggregate >> > option that includes >> > the platform default. I'd prefer not to have any statefulness of the >> > core.fsync setting so >> > that there is less confusion about the final fsync configuration. >> >> Then scratch your preference ;-) > > Just to clarify, linguistically, by 'scratch' do you mean that I should drop > my preference Yes. > Is there a well-defined place where we know that configuration processing > is complete? The most obvious spot to me to integrate these two values would > be the first time we need to figure out the fsync state. That sounds like a good place. ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v5 3/5] core.fsync: introduce granular fsync control 2022-03-10 18:44 ` Junio C Hamano @ 2022-03-10 19:57 ` Junio C Hamano 2022-03-10 20:25 ` Neeraj Singh 0 siblings, 1 reply; 122+ messages in thread From: Junio C Hamano @ 2022-03-10 19:57 UTC (permalink / raw) To: Neeraj Singh Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Ævar Arnfjörð Bjarmason, Patrick Steinhardt, brian m. carlson, Neeraj K. Singh Junio C Hamano <gitster@pobox.com> writes: > Neeraj Singh <nksingh85@gmail.com> writes: > >>> > At the conclusion of this series, I defined 'default' as an aggregate >>> > option that includes >>> > the platform default. I'd prefer not to have any statefulness of the >>> > core.fsync setting so >>> > that there is less confusion about the final fsync configuration. >>> >>> Then scratch your preference ;-) >> >> Just to clarify, linguistically, by 'scratch' do you mean that I should drop >> my preference > > Yes. Let me take this part back. I do not mind too deeply if this were "each occurrence of core.fsync as a whole replaces whatever we saw earlier, i.e. last-one-wins". But if we were going that route, instead of starting from an empty set, I'd prefer to see it begin with the built-in default (i.e. the one you defined to mimic the traditional behaviour before core.fsync was introduced) and added or deleted by each (possibly '-' prefixed) element on the comma-separated list, with an explicit way to clear the built-in default. E.g. "none,refs" would clear the components traditionally fsync'ed by default and choose only "refs" component, while "-pack-metadata" would mean the default ones minus "pack-metadata" component are subject for fsync'ing. An empty string would naturally mean "By having this core.fsync entry, I am telling you not to pay any attention to what lower-precedence configuration files said. But I want the built-in default, without any additions or subtractions made by this entry, just the default, please" in such a scheme, so do not forbid it. Or, we can inherit from the previous configuration file to allow /etc/gitconfig and the ones shipped by Git for Windows to augment the built-in default before letting end-user configuration to further customize the preference. Either is fine by me. Thanks. ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v5 3/5] core.fsync: introduce granular fsync control 2022-03-10 19:57 ` Junio C Hamano @ 2022-03-10 20:25 ` Neeraj Singh 2022-03-10 21:17 ` Junio C Hamano 0 siblings, 1 reply; 122+ messages in thread From: Neeraj Singh @ 2022-03-10 20:25 UTC (permalink / raw) To: Junio C Hamano Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Ævar Arnfjörð Bjarmason, Patrick Steinhardt, brian m. carlson, Neeraj K. Singh On Thu, Mar 10, 2022 at 11:57 AM Junio C Hamano <gitster@pobox.com> wrote: > > Junio C Hamano <gitster@pobox.com> writes: > > > Neeraj Singh <nksingh85@gmail.com> writes: > > > >>> > At the conclusion of this series, I defined 'default' as an aggregate > >>> > option that includes > >>> > the platform default. I'd prefer not to have any statefulness of the > >>> > core.fsync setting so > >>> > that there is less confusion about the final fsync configuration. > >>> > >>> Then scratch your preference ;-) > >> > >> Just to clarify, linguistically, by 'scratch' do you mean that I should drop > >> my preference > > > > Yes. > > Let me take this part back. > > I do not mind too deeply if this were "each occurrence of core.fsync > as a whole replaces whatever we saw earlier, i.e. last-one-wins". > > But if we were going that route, instead of starting from an empty > set, I'd prefer to see it begin with the built-in default (i.e. the > one you defined to mimic the traditional behaviour before core.fsync > was introduced) and added or deleted by each (possibly '-' prefixed) > element on the comma-separated list, with an explicit way to clear > the built-in default. E.g. "none,refs" would clear the components > traditionally fsync'ed by default and choose only "refs" component, > while "-pack-metadata" would mean the default ones minus > "pack-metadata" component are subject for fsync'ing. An empty > string would naturally mean "By having this core.fsync entry, I am > telling you not to pay any attention to what lower-precedence > configuration files said. But I want the built-in default, without > any additions or subtractions made by this entry, just the default, > please" in such a scheme, so do not forbid it. > > Or, we can inherit from the previous configuration file to allow > /etc/gitconfig and the ones shipped by Git for Windows to augment > the built-in default before letting end-user configuration to > further customize the preference. > > Either is fine by me. > > Thanks. Okay, I'll implement this version, since this is close to my preference. Under this schema, 'default' isn't useful as an aggregate option, so I'll eliminate that. ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v5 3/5] core.fsync: introduce granular fsync control 2022-03-10 20:25 ` Neeraj Singh @ 2022-03-10 21:17 ` Junio C Hamano 0 siblings, 0 replies; 122+ messages in thread From: Junio C Hamano @ 2022-03-10 21:17 UTC (permalink / raw) To: Neeraj Singh Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Ævar Arnfjörð Bjarmason, Patrick Steinhardt, brian m. carlson, Neeraj K. Singh Neeraj Singh <nksingh85@gmail.com> writes: > Under this schema, 'default' isn't useful as an aggregate option, so > I'll eliminate > that. Yeah, the only difference is the starting point. Either start with default set of bits and give an option to clear, or start with an empty set and give an option to set default. The former may be a bit less cumbersome to users but there isn't a huge difference. ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v5 3/5] core.fsync: introduce granular fsync control 2022-03-10 2:53 ` Neeraj Singh 2022-03-10 7:19 ` Junio C Hamano @ 2022-03-10 13:11 ` Johannes Schindelin 2022-03-10 17:18 ` Junio C Hamano 2 siblings, 0 replies; 122+ messages in thread From: Johannes Schindelin @ 2022-03-10 13:11 UTC (permalink / raw) To: Neeraj Singh Cc: Junio C Hamano, Neeraj Singh via GitGitGadget, Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Ævar Arnfjörð Bjarmason, Patrick Steinhardt, brian m. carlson, Neeraj K. Singh Hi Neeraj, On Wed, 9 Mar 2022, Neeraj Singh wrote: > On Wed, Mar 9, 2022 at 4:21 PM Junio C Hamano <gitster@pobox.com> wrote: > > > I am wondering if fsync_or_die() interface is abstracted well enough, > > or we need things like "the fd is inside this directory; in addition > > to doing the fsync of the fd, please sync the parent directory as > > well" support before we start adding more components (if there is such > > a need, perhaps it comes before this step). > > > > I think syncing the parent directory is a separate fsyncMethod that > would require changes across the codebase to obtain an appropriate > directory fd. I'd prefer to treat that as a separable concern. That makes sense to me because I expect further abstraction to be necessary here because Unix/Linux semantics differ quite a bit more from Windows semantics when it comes to directory "file" descriptors than when talking about files' file descriptors. > > > +#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \ > > > + FSYNC_COMPONENT_PACK_METADATA | \ > > > + FSYNC_COMPONENT_COMMIT_GRAPH) > > > > IOW, everything other than loose object, which already has a > > separate core.fsyncObjectFiles knob to loosen. Everything else we > > currently sync unconditionally and the default keeps that > > arrangement? > > > > Yes, trying to keep default behavior identical on non-Windows > platforms. Windows will be expected to adopt batch mode, and have > loose objects in this set. We already adopted an early version of this patch series: https://github.com/git-for-windows/git/commit/98209a5f6e4 And yes, I will gladly adapt that to whatever lands in core Git. Thank you _so_ much for working on this! Dscho ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v5 3/5] core.fsync: introduce granular fsync control 2022-03-10 2:53 ` Neeraj Singh 2022-03-10 7:19 ` Junio C Hamano 2022-03-10 13:11 ` Johannes Schindelin @ 2022-03-10 17:18 ` Junio C Hamano 2 siblings, 0 replies; 122+ messages in thread From: Junio C Hamano @ 2022-03-10 17:18 UTC (permalink / raw) To: Neeraj Singh Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Ævar Arnfjörð Bjarmason, Patrick Steinhardt, brian m. carlson, Neeraj K. Singh Neeraj Singh <nksingh85@gmail.com> writes: >> I am wondering if fsync_or_die() interface is abstracted well >> enough, or we need things like "the fd is inside this directory; in >> addition to doing the fsync of the fd, please sync the parent >> directory as well" support before we start adding more components >> (if there is such a need, perhaps it comes before this step). >> > > I think syncing the parent directory is a separate fsyncMethod that > would require changes across the codebase to obtain an appropriate > directory fd. I'd prefer to treat that as a separable concern. Yeah, that would be a sensible direction to go. If we never did the "sync the parent" thing, we do not need it in the fsyncMethod world immediately. It can be added later. Thanks. ^ permalink raw reply [flat|nested] 122+ messages in thread
* [PATCH v5 4/5] core.fsync: new option to harden the index 2022-03-09 23:03 ` [PATCH v5 0/5] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget ` (2 preceding siblings ...) 2022-03-09 23:03 ` [PATCH v5 3/5] core.fsync: introduce granular fsync control Neeraj Singh via GitGitGadget @ 2022-03-09 23:03 ` Neeraj Singh via GitGitGadget 2022-03-09 23:03 ` [PATCH v5 5/5] core.fsync: documentation and user-friendly aggregate options Neeraj Singh via GitGitGadget ` (5 subsequent siblings) 9 siblings, 0 replies; 122+ messages in thread From: Neeraj Singh via GitGitGadget @ 2022-03-09 23:03 UTC (permalink / raw) To: git Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps, sandals, Neeraj K. Singh, Neeraj Singh From: Neeraj Singh <neerajsi@microsoft.com> This commit introduces the new ability for the user to harden the index. In the event of a system crash, the index must be durable for the user to actually find a file that has been added to the repo and then deleted from the working tree. We use the presence of the COMMIT_LOCK flag and absence of the alternate_index_output as a proxy for determining whether we're updating the persistent index of the repo or some temporary index. We don't sync these temporary indexes. Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> --- cache.h | 1 + config.c | 1 + read-cache.c | 19 +++++++++++++------ 3 files changed, 15 insertions(+), 6 deletions(-) diff --git a/cache.h b/cache.h index a5eaa60a7a8..3fefe8f6f60 100644 --- a/cache.h +++ b/cache.h @@ -1004,6 +1004,7 @@ enum fsync_component { FSYNC_COMPONENT_PACK = 1 << 1, FSYNC_COMPONENT_PACK_METADATA = 1 << 2, FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3, + FSYNC_COMPONENT_INDEX = 1 << 4, }; #define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \ diff --git a/config.c b/config.c index 51a35715642..bb44f6f506d 100644 --- a/config.c +++ b/config.c @@ -1331,6 +1331,7 @@ static const struct fsync_component_entry { { "pack", FSYNC_COMPONENT_PACK }, { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA }, { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, + { "index", FSYNC_COMPONENT_INDEX }, }; static enum fsync_component parse_fsync_components(const char *var, const char *string) diff --git a/read-cache.c b/read-cache.c index df869691fd4..7683b679258 100644 --- a/read-cache.c +++ b/read-cache.c @@ -2842,7 +2842,7 @@ static int record_ieot(void) * rely on it. */ static int do_write_index(struct index_state *istate, struct tempfile *tempfile, - int strip_extensions) + int strip_extensions, unsigned flags) { uint64_t start = getnanotime(); struct hashfile *f; @@ -2856,6 +2856,7 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile, struct strbuf previous_name_buf = STRBUF_INIT, *previous_name; int drop_cache_tree = istate->drop_cache_tree; off_t offset; + int csum_fsync_flag; int ieot_entries = 1; struct index_entry_offset_table *ieot = NULL; int nr, nr_threads; @@ -3089,7 +3090,13 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile, return -1; } - finalize_hashfile(f, istate->oid.hash, FSYNC_COMPONENT_NONE, CSUM_HASH_IN_STREAM); + csum_fsync_flag = 0; + if (!alternate_index_output && (flags & COMMIT_LOCK)) + csum_fsync_flag = CSUM_FSYNC; + + finalize_hashfile(f, istate->oid.hash, FSYNC_COMPONENT_INDEX, + CSUM_HASH_IN_STREAM | csum_fsync_flag); + if (close_tempfile_gently(tempfile)) { error(_("could not close '%s'"), get_tempfile_path(tempfile)); return -1; @@ -3144,7 +3151,7 @@ static int do_write_locked_index(struct index_state *istate, struct lock_file *l */ trace2_region_enter_printf("index", "do_write_index", the_repository, "%s", get_lock_file_path(lock)); - ret = do_write_index(istate, lock->tempfile, 0); + ret = do_write_index(istate, lock->tempfile, 0, flags); trace2_region_leave_printf("index", "do_write_index", the_repository, "%s", get_lock_file_path(lock)); @@ -3238,7 +3245,7 @@ static int clean_shared_index_files(const char *current_hex) } static int write_shared_index(struct index_state *istate, - struct tempfile **temp) + struct tempfile **temp, unsigned flags) { struct split_index *si = istate->split_index; int ret, was_full = !istate->sparse_index; @@ -3248,7 +3255,7 @@ static int write_shared_index(struct index_state *istate, trace2_region_enter_printf("index", "shared/do_write_index", the_repository, "%s", get_tempfile_path(*temp)); - ret = do_write_index(si->base, *temp, 1); + ret = do_write_index(si->base, *temp, 1, flags); trace2_region_leave_printf("index", "shared/do_write_index", the_repository, "%s", get_tempfile_path(*temp)); @@ -3357,7 +3364,7 @@ int write_locked_index(struct index_state *istate, struct lock_file *lock, ret = do_write_locked_index(istate, lock, flags); goto out; } - ret = write_shared_index(istate, &temp); + ret = write_shared_index(istate, &temp, flags); saved_errno = errno; if (is_tempfile_active(temp)) -- gitgitgadget ^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v5 5/5] core.fsync: documentation and user-friendly aggregate options 2022-03-09 23:03 ` [PATCH v5 0/5] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget ` (3 preceding siblings ...) 2022-03-09 23:03 ` [PATCH v5 4/5] core.fsync: new option to harden the index Neeraj Singh via GitGitGadget @ 2022-03-09 23:03 ` Neeraj Singh via GitGitGadget 2022-03-10 9:53 ` Future-proofed syncing of refs Patrick Steinhardt ` (4 subsequent siblings) 9 siblings, 0 replies; 122+ messages in thread From: Neeraj Singh via GitGitGadget @ 2022-03-09 23:03 UTC (permalink / raw) To: git Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps, sandals, Neeraj K. Singh, Neeraj Singh From: Neeraj Singh <neerajsi@microsoft.com> This commit adds aggregate options for the core.fsync setting that are more user-friendly. These options are specified in terms of 'levels of safety', indicating which Git operations are considered to be sync points for durability. The new documentation is also included here in its entirety for ease of review. Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> --- Documentation/config/core.txt | 36 +++++++++++++++++++++++++++++++++++ cache.h | 23 +++++++++++++++++++--- config.c | 6 ++++++ 3 files changed, 62 insertions(+), 3 deletions(-) diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt index 74399072843..973805e8a98 100644 --- a/Documentation/config/core.txt +++ b/Documentation/config/core.txt @@ -547,6 +547,42 @@ core.whitespace:: is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent` errors. The default tab width is 8. Allowed values are 1 to 63. +core.fsync:: + A comma-separated list of parts of the repository which should be + hardened via the core.fsyncMethod when created or modified. You can + disable hardening of any component by prefixing it with a '-'. Later + items take precedence over earlier ones in the comma-separated list. + For example, `core.fsync=all,-pack-metadata` means "harden everything + except pack metadata." Items that are not hardened may be lost in the + event of an unclean system shutdown. Unless you have special + requirements, it is recommended that you leave this option as default + or pick one of `committed`, `added`, or `all`. ++ +* `none` disables fsync completely. This value must be specified alone. +* `loose-object` hardens objects added to the repo in loose-object form. +* `pack` hardens objects added to the repo in packfile form. +* `pack-metadata` hardens packfile bitmaps and indexes. +* `commit-graph` hardens the commit graph file. +* `index` hardens the index when it is modified. +* `objects` is an aggregate option that is equivalent to + `loose-object,pack`. +* `derived-metadata` is an aggregate option that is equivalent to + `pack-metadata,commit-graph`. +* `default` is an aggregate option that is equivalent to + `objects,derived-metadata,-loose-object`. This mode is enabled by default. + It has good performance, but risks losing recent work if the system shuts + down uncleanly, since commits, trees, and blobs in loose-object form may be + lost. +* `committed` is an aggregate option that is currently equivalent to + `objects`. This mode sacrifices some performance to ensure that all work + that is committed to the repository with `git commit` or similar commands + is preserved. +* `added` is an aggregate option that is currently equivalent to + `committed,index`. This mode sacrifices additional performance to + ensure that the results of commands like `git add` and similar operations + are preserved. +* `all` is an aggregate option that syncs all individual components above. + core.fsyncMethod:: A value indicating the strategy Git will use to harden repository data using fsync and related primitives. diff --git a/cache.h b/cache.h index 3fefe8f6f60..833f0236e68 100644 --- a/cache.h +++ b/cache.h @@ -1007,9 +1007,26 @@ enum fsync_component { FSYNC_COMPONENT_INDEX = 1 << 4, }; -#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \ - FSYNC_COMPONENT_PACK_METADATA | \ - FSYNC_COMPONENT_COMMIT_GRAPH) +#define FSYNC_COMPONENTS_OBJECTS (FSYNC_COMPONENT_LOOSE_OBJECT | \ + FSYNC_COMPONENT_PACK) + +#define FSYNC_COMPONENTS_DERIVED_METADATA (FSYNC_COMPONENT_PACK_METADATA | \ + FSYNC_COMPONENT_COMMIT_GRAPH) + +#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENTS_OBJECTS | \ + FSYNC_COMPONENTS_DERIVED_METADATA | \ + ~FSYNC_COMPONENT_LOOSE_OBJECT) + +#define FSYNC_COMPONENTS_COMMITTED (FSYNC_COMPONENTS_OBJECTS) + +#define FSYNC_COMPONENTS_ADDED (FSYNC_COMPONENTS_COMMITTED | \ + FSYNC_COMPONENT_INDEX) + +#define FSYNC_COMPONENTS_ALL (FSYNC_COMPONENT_LOOSE_OBJECT | \ + FSYNC_COMPONENT_PACK | \ + FSYNC_COMPONENT_PACK_METADATA | \ + FSYNC_COMPONENT_COMMIT_GRAPH | \ + FSYNC_COMPONENT_INDEX) /* * A bitmask indicating which components of the repo should be fsynced. diff --git a/config.c b/config.c index bb44f6f506d..3976ec74fd4 100644 --- a/config.c +++ b/config.c @@ -1332,6 +1332,12 @@ static const struct fsync_component_entry { { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA }, { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, { "index", FSYNC_COMPONENT_INDEX }, + { "objects", FSYNC_COMPONENTS_OBJECTS }, + { "derived-metadata", FSYNC_COMPONENTS_DERIVED_METADATA }, + { "default", FSYNC_COMPONENTS_DEFAULT }, + { "committed", FSYNC_COMPONENTS_COMMITTED }, + { "added", FSYNC_COMPONENTS_ADDED }, + { "all", FSYNC_COMPONENTS_ALL }, }; static enum fsync_component parse_fsync_components(const char *var, const char *string) -- gitgitgadget ^ permalink raw reply related [flat|nested] 122+ messages in thread
* Future-proofed syncing of refs 2022-03-09 23:03 ` [PATCH v5 0/5] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget ` (4 preceding siblings ...) 2022-03-09 23:03 ` [PATCH v5 5/5] core.fsync: documentation and user-friendly aggregate options Neeraj Singh via GitGitGadget @ 2022-03-10 9:53 ` Patrick Steinhardt 2022-03-10 9:53 ` [PATCH 6/8] core.fsync: add `fsync_component()` wrapper which doesn't die Patrick Steinhardt ` (3 subsequent siblings) 9 siblings, 0 replies; 122+ messages in thread From: Patrick Steinhardt @ 2022-03-10 9:53 UTC (permalink / raw) To: git Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, sandals, Neeraj K. Singh, Junio C Hamano [-- Attachment #1: Type: text/plain, Size: 1472 bytes --] Hi, these three patches apply on top of Neeraj's v5 of his "A design for future-proofing fsync() configuration". I'm sending this as a reply to his v5 to keep the discussion in one place -- I think ultimately, we may want to merge both series into a single one anyway. But please shout at me if this is considered bad style and I'll split it out into a separate thread. In any case, these three patches implement fsyncing for loose and packed references using the proposed `core.fsync` option, with three additional knobs: - "loose-ref" will fsync loose references. - "packed-refs" will fsync packed references. - "refs" will fsync all references, which should ideally also include all new backends like the reftable backend. I think this extension demonstrates that the design proposed by Neeraj is quite easy to extend without too much boilerplate. Patrick Patrick Steinhardt (3): core.fsync: add `fsync_component()` wrapper which doesn't die core.fsync: new option to harden loose references core.fsync: new option to harden packed references Documentation/config/core.txt | 3 +++ cache.h | 22 ++++++++++++++++++---- config.c | 3 +++ refs/files-backend.c | 29 +++++++++++++++++++++++++++++ refs/packed-backend.c | 3 ++- write-or-die.c | 10 ++++++---- 6 files changed, 61 insertions(+), 9 deletions(-) -- 2.35.1 [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 122+ messages in thread
* [PATCH 6/8] core.fsync: add `fsync_component()` wrapper which doesn't die 2022-03-09 23:03 ` [PATCH v5 0/5] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget ` (5 preceding siblings ...) 2022-03-10 9:53 ` Future-proofed syncing of refs Patrick Steinhardt @ 2022-03-10 9:53 ` Patrick Steinhardt 2022-03-10 17:34 ` Junio C Hamano 2022-03-10 9:53 ` [PATCH 7/8] core.fsync: new option to harden loose references Patrick Steinhardt ` (2 subsequent siblings) 9 siblings, 1 reply; 122+ messages in thread From: Patrick Steinhardt @ 2022-03-10 9:53 UTC (permalink / raw) To: git Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, sandals, Neeraj K. Singh, Junio C Hamano [-- Attachment #1: Type: text/plain, Size: 2142 bytes --] We have a `fsync_component_or_die()` helper function which only syncs changes to disk in case the corresponding config is enabled by the user. This wrapper will always die on an error though, which makes it insufficient for new callsites we are about to add. Add a new `fsync_component()` wrapper which returns an error code instead of dying. Signed-off-by: Patrick Steinhardt <ps@pks.im> --- cache.h | 13 ++++++++++--- write-or-die.c | 10 ++++++---- 2 files changed, 16 insertions(+), 7 deletions(-) diff --git a/cache.h b/cache.h index f307e89516..63a95d1977 100644 --- a/cache.h +++ b/cache.h @@ -1745,12 +1745,19 @@ int copy_file(const char *dst, const char *src, int mode); int copy_file_with_time(const char *dst, const char *src, int mode); void write_or_die(int fd, const void *buf, size_t count); -void fsync_or_die(int fd, const char *); +int maybe_fsync(int fd); + +static inline int fsync_component(enum fsync_component component, int fd) +{ + if (fsync_components & component) + return maybe_fsync(fd); + return 0; +} static inline void fsync_component_or_die(enum fsync_component component, int fd, const char *msg) { - if (fsync_components & component) - fsync_or_die(fd, msg); + if (fsync_component(component, fd) < 0) + die_errno("fsync error on '%s'", msg); } ssize_t read_in_full(int fd, void *buf, size_t count); diff --git a/write-or-die.c b/write-or-die.c index 9faa5f9f56..4a5455ce46 100644 --- a/write-or-die.c +++ b/write-or-die.c @@ -56,19 +56,21 @@ void fprintf_or_die(FILE *f, const char *fmt, ...) } } -void fsync_or_die(int fd, const char *msg) +int maybe_fsync(int fd) { if (use_fsync < 0) use_fsync = git_env_bool("GIT_TEST_FSYNC", 1); if (!use_fsync) - return; + return 0; if (fsync_method == FSYNC_METHOD_WRITEOUT_ONLY && git_fsync(fd, FSYNC_WRITEOUT_ONLY) >= 0) - return; + return 0; if (git_fsync(fd, FSYNC_HARDWARE_FLUSH) < 0) - die_errno("fsync error on '%s'", msg); + return -1; + + return 0; } void write_or_die(int fd, const void *buf, size_t count) -- 2.35.1 [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply related [flat|nested] 122+ messages in thread
* Re: [PATCH 6/8] core.fsync: add `fsync_component()` wrapper which doesn't die 2022-03-10 9:53 ` [PATCH 6/8] core.fsync: add `fsync_component()` wrapper which doesn't die Patrick Steinhardt @ 2022-03-10 17:34 ` Junio C Hamano 2022-03-10 18:40 ` Neeraj Singh 0 siblings, 1 reply; 122+ messages in thread From: Junio C Hamano @ 2022-03-10 17:34 UTC (permalink / raw) To: Patrick Steinhardt Cc: git, rsbecker, bagasdotme, newren, avarab, nksingh85, sandals, Neeraj K. Singh Patrick Steinhardt <ps@pks.im> writes: > We have a `fsync_component_or_die()` helper function which only syncs > changes to disk in case the corresponding config is enabled by the user. > This wrapper will always die on an error though, which makes it > insufficient for new callsites we are about to add. You can replace "which makes it ..." part with a bit more concrete description to save suspense from the readers. fsync_component_or_die() that dies upon an error is not useful for callers with their own error handling or recovery logic, like ref transaction API. Split fsync_component() out that returns an error to help them. > -void fsync_or_die(int fd, const char *); > +int maybe_fsync(int fd); > ... > +static inline int fsync_component(enum fsync_component component, int fd) > +{ > + if (fsync_components & component) > + return maybe_fsync(fd); > + return 0; > +} > > static inline void fsync_component_or_die(enum fsync_component component, int fd, const char *msg) > { > - if (fsync_components & component) > - fsync_or_die(fd, msg); > + if (fsync_component(component, fd) < 0) > + die_errno("fsync error on '%s'", msg); > } I think in the eventuall reroll, these "static inline" functions on the I/O code path will become real functions in write-or-die.c but other than that this reorganization looks sensible. Thanks. > diff --git a/write-or-die.c b/write-or-die.c > index 9faa5f9f56..4a5455ce46 100644 > --- a/write-or-die.c > +++ b/write-or-die.c > @@ -56,19 +56,21 @@ void fprintf_or_die(FILE *f, const char *fmt, ...) > } > } > > -void fsync_or_die(int fd, const char *msg) > +int maybe_fsync(int fd) > { > if (use_fsync < 0) > use_fsync = git_env_bool("GIT_TEST_FSYNC", 1); > if (!use_fsync) > - return; > + return 0; > > if (fsync_method == FSYNC_METHOD_WRITEOUT_ONLY && > git_fsync(fd, FSYNC_WRITEOUT_ONLY) >= 0) > - return; > + return 0; > > if (git_fsync(fd, FSYNC_HARDWARE_FLUSH) < 0) > - die_errno("fsync error on '%s'", msg); > + return -1; > + > + return 0; > } > > void write_or_die(int fd, const void *buf, size_t count) ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH 6/8] core.fsync: add `fsync_component()` wrapper which doesn't die 2022-03-10 17:34 ` Junio C Hamano @ 2022-03-10 18:40 ` Neeraj Singh 0 siblings, 0 replies; 122+ messages in thread From: Neeraj Singh @ 2022-03-10 18:40 UTC (permalink / raw) To: Junio C Hamano Cc: Patrick Steinhardt, Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Ævar Arnfjörð Bjarmason, brian m. carlson, Neeraj K. Singh On Thu, Mar 10, 2022 at 9:34 AM Junio C Hamano <gitster@pobox.com> wrote: > > Patrick Steinhardt <ps@pks.im> writes: > > > We have a `fsync_component_or_die()` helper function which only syncs > > changes to disk in case the corresponding config is enabled by the user. > > This wrapper will always die on an error though, which makes it > > insufficient for new callsites we are about to add. > > You can replace "which makes it ..." part with a bit more concrete > description to save suspense from the readers. > > fsync_component_or_die() that dies upon an error is not useful > for callers with their own error handling or recovery logic, > like ref transaction API. > > Split fsync_component() out that returns an error to help them. > > > -void fsync_or_die(int fd, const char *); > > +int maybe_fsync(int fd); > > ... > > +static inline int fsync_component(enum fsync_component component, int fd) > > +{ > > + if (fsync_components & component) > > + return maybe_fsync(fd); > > + return 0; > > +} > > > > static inline void fsync_component_or_die(enum fsync_component component, int fd, const char *msg) > > { > > - if (fsync_components & component) > > - fsync_or_die(fd, msg); > > + if (fsync_component(component, fd) < 0) > > + die_errno("fsync error on '%s'", msg); > > } > > I think in the eventuall reroll, these "static inline" functions on > the I/O code path will become real functions in write-or-die.c but > other than that this reorganization looks sensible. > > Thanks. > Yes, that will be part of v6 of the base changeset. > > diff --git a/write-or-die.c b/write-or-die.c > > index 9faa5f9f56..4a5455ce46 100644 > > --- a/write-or-die.c > > +++ b/write-or-die.c > > @@ -56,19 +56,21 @@ void fprintf_or_die(FILE *f, const char *fmt, ...) > > } > > } > > > > -void fsync_or_die(int fd, const char *msg) > > +int maybe_fsync(int fd) > > { > > if (use_fsync < 0) > > use_fsync = git_env_bool("GIT_TEST_FSYNC", 1); > > if (!use_fsync) > > - return; > > + return 0; > > > > if (fsync_method == FSYNC_METHOD_WRITEOUT_ONLY && > > git_fsync(fd, FSYNC_WRITEOUT_ONLY) >= 0) > > - return; > > + return 0; > > > > if (git_fsync(fd, FSYNC_HARDWARE_FLUSH) < 0) > > - die_errno("fsync error on '%s'", msg); > > + return -1; > > + > > + return 0; > > } > > > > void write_or_die(int fd, const void *buf, size_t count) ^ permalink raw reply [flat|nested] 122+ messages in thread
* [PATCH 7/8] core.fsync: new option to harden loose references 2022-03-09 23:03 ` [PATCH v5 0/5] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget ` (6 preceding siblings ...) 2022-03-10 9:53 ` [PATCH 6/8] core.fsync: add `fsync_component()` wrapper which doesn't die Patrick Steinhardt @ 2022-03-10 9:53 ` Patrick Steinhardt 2022-03-10 18:25 ` Junio C Hamano ` (2 more replies) 2022-03-10 9:53 ` [PATCH 8/8] core.fsync: new option to harden packed references Patrick Steinhardt 2022-03-10 22:43 ` [PATCH v6 0/6] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget 9 siblings, 3 replies; 122+ messages in thread From: Patrick Steinhardt @ 2022-03-10 9:53 UTC (permalink / raw) To: git Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, sandals, Neeraj K. Singh, Junio C Hamano [-- Attachment #1: Type: text/plain, Size: 6283 bytes --] When writing loose references to disk we first create a lockfile, write the updated value of the reference into that lockfile, and on commit we rename the file into place. According to filesystem developers, this behaviour is broken because applications should always sync data to disk before doing the final rename to ensure data consistency [1][2][3]. If applications fail to do this correctly, a hard crash of the machine can easily result in corrupted on-disk data. This kind of corruption can in fact be easily observed with Git when the machine hard-crashes shortly after writing loose references to disk. On machines with ext4, this will likely lead to the "empty files" problem: the file has been renamed, but its data has not been synced to disk. The result is that the references is corrupt, and in the worst case this can lead to data loss. Implement a new option to harden loose references so that users and admins can avoid this scenario by syncing locked loose references to disk before we rename them into place. [1]: https://thunk.org/tytso/blog/2009/03/15/dont-fear-the-fsync/ [2]: https://btrfs.wiki.kernel.org/index.php/FAQ (What are the crash guarantees of overwrite-by-rename) [3]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/admin-guide/ext4.rst (see auto_da_alloc) Signed-off-by: Patrick Steinhardt <ps@pks.im> --- Documentation/config/core.txt | 2 ++ cache.h | 6 +++++- config.c | 2 ++ refs/files-backend.c | 29 +++++++++++++++++++++++++++++ 4 files changed, 38 insertions(+), 1 deletion(-) diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt index 973805e8a9..b67d3c340e 100644 --- a/Documentation/config/core.txt +++ b/Documentation/config/core.txt @@ -564,8 +564,10 @@ core.fsync:: * `pack-metadata` hardens packfile bitmaps and indexes. * `commit-graph` hardens the commit graph file. * `index` hardens the index when it is modified. +* `loose-ref` hardens references modified in the repo in loose-ref form. * `objects` is an aggregate option that is equivalent to `loose-object,pack`. +* `refs` is an aggregate option that is equivalent to `loose-ref`. * `derived-metadata` is an aggregate option that is equivalent to `pack-metadata,commit-graph`. * `default` is an aggregate option that is equivalent to diff --git a/cache.h b/cache.h index 63a95d1977..b56a56f539 100644 --- a/cache.h +++ b/cache.h @@ -1005,11 +1005,14 @@ enum fsync_component { FSYNC_COMPONENT_PACK_METADATA = 1 << 2, FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3, FSYNC_COMPONENT_INDEX = 1 << 4, + FSYNC_COMPONENT_LOOSE_REF = 1 << 5, }; #define FSYNC_COMPONENTS_OBJECTS (FSYNC_COMPONENT_LOOSE_OBJECT | \ FSYNC_COMPONENT_PACK) +#define FSYNC_COMPONENTS_REFS (FSYNC_COMPONENT_LOOSE_REF) + #define FSYNC_COMPONENTS_DERIVED_METADATA (FSYNC_COMPONENT_PACK_METADATA | \ FSYNC_COMPONENT_COMMIT_GRAPH) @@ -1026,7 +1029,8 @@ enum fsync_component { FSYNC_COMPONENT_PACK | \ FSYNC_COMPONENT_PACK_METADATA | \ FSYNC_COMPONENT_COMMIT_GRAPH | \ - FSYNC_COMPONENT_INDEX) + FSYNC_COMPONENT_INDEX | \ + FSYNC_COMPONENT_LOOSE_REF) /* * A bitmask indicating which components of the repo should be fsynced. diff --git a/config.c b/config.c index f03f27c3de..b5d3e6e404 100644 --- a/config.c +++ b/config.c @@ -1332,7 +1332,9 @@ static const struct fsync_component_entry { { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA }, { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, { "index", FSYNC_COMPONENT_INDEX }, + { "loose-ref", FSYNC_COMPONENT_LOOSE_REF }, { "objects", FSYNC_COMPONENTS_OBJECTS }, + { "refs", FSYNC_COMPONENTS_REFS }, { "derived-metadata", FSYNC_COMPONENTS_DERIVED_METADATA }, { "default", FSYNC_COMPONENTS_DEFAULT }, { "committed", FSYNC_COMPONENTS_COMMITTED }, diff --git a/refs/files-backend.c b/refs/files-backend.c index f59589d6cc..279316de45 100644 --- a/refs/files-backend.c +++ b/refs/files-backend.c @@ -1392,6 +1392,15 @@ static int refs_rename_ref_available(struct ref_store *refs, return ok; } +static int files_sync_loose_ref(struct ref_lock *lock, struct strbuf *err) +{ + int ret = fsync_component(FSYNC_COMPONENT_LOOSE_REF, get_lock_file_fd(&lock->lk)); + if (ret) + strbuf_addf(err, "could not sync loose ref '%s': %s", lock->ref_name, + strerror(errno)); + return ret; +} + static int files_copy_or_rename_ref(struct ref_store *ref_store, const char *oldrefname, const char *newrefname, const char *logmsg, int copy) @@ -1504,6 +1513,7 @@ static int files_copy_or_rename_ref(struct ref_store *ref_store, oidcpy(&lock->old_oid, &orig_oid); if (write_ref_to_lockfile(lock, &orig_oid, 0, &err) || + files_sync_loose_ref(lock, &err) || commit_ref_update(refs, lock, &orig_oid, logmsg, &err)) { error("unable to write current sha1 into %s: %s", newrefname, err.buf); strbuf_release(&err); @@ -1524,6 +1534,7 @@ static int files_copy_or_rename_ref(struct ref_store *ref_store, flag = log_all_ref_updates; log_all_ref_updates = LOG_REFS_NONE; if (write_ref_to_lockfile(lock, &orig_oid, 0, &err) || + files_sync_loose_ref(lock, &err) || commit_ref_update(refs, lock, &orig_oid, NULL, &err)) { error("unable to write current sha1 into %s: %s", oldrefname, err.buf); strbuf_release(&err); @@ -2819,6 +2830,24 @@ static int files_transaction_prepare(struct ref_store *ref_store, } } + /* + * Sync all lockfiles to disk to ensure data consistency. We do this in + * a separate step such that we can sync all modified refs in a single + * step, which may be more efficient on some filesystems. + */ + for (i = 0; i < transaction->nr; i++) { + struct ref_update *update = transaction->updates[i]; + struct ref_lock *lock = update->backend_data; + + if (!(update->flags & REF_NEEDS_COMMIT)) + continue; + + if (files_sync_loose_ref(lock, err)) { + ret = TRANSACTION_GENERIC_ERROR; + goto cleanup; + } + } + cleanup: free(head_ref); string_list_clear(&affected_refnames, 0); -- 2.35.1 [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply related [flat|nested] 122+ messages in thread
* Re: [PATCH 7/8] core.fsync: new option to harden loose references 2022-03-10 9:53 ` [PATCH 7/8] core.fsync: new option to harden loose references Patrick Steinhardt @ 2022-03-10 18:25 ` Junio C Hamano 2022-03-10 19:03 ` Neeraj Singh 2022-03-10 22:54 ` Neeraj Singh 2022-03-11 6:40 ` Junio C Hamano 2 siblings, 1 reply; 122+ messages in thread From: Junio C Hamano @ 2022-03-10 18:25 UTC (permalink / raw) To: Patrick Steinhardt Cc: git, rsbecker, bagasdotme, newren, avarab, nksingh85, sandals, Neeraj K. Singh Patrick Steinhardt <ps@pks.im> writes: > diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt > index 973805e8a9..b67d3c340e 100644 > --- a/Documentation/config/core.txt > +++ b/Documentation/config/core.txt > @@ -564,8 +564,10 @@ core.fsync:: > * `pack-metadata` hardens packfile bitmaps and indexes. > * `commit-graph` hardens the commit graph file. > * `index` hardens the index when it is modified. > +* `loose-ref` hardens references modified in the repo in loose-ref form. > * `objects` is an aggregate option that is equivalent to > `loose-object,pack`. > +* `refs` is an aggregate option that is equivalent to `loose-ref`. Aggregate of one feels strange. I do not see a strong reason to have two separate classes loose vs packed and allow them be protected to different robustness, but that aside, if we were to have two, this "aggregate" is better added when the second one is. Having said that, given that a separate ref backend that has no distinction between loose or packed is on horizen, I think we would rather prefer to see a single "ref" component that governs all backends. > @@ -1026,7 +1029,8 @@ enum fsync_component { > FSYNC_COMPONENT_PACK | \ > FSYNC_COMPONENT_PACK_METADATA | \ > FSYNC_COMPONENT_COMMIT_GRAPH | \ > - FSYNC_COMPONENT_INDEX) > + FSYNC_COMPONENT_INDEX | \ > + FSYNC_COMPONENT_LOOSE_REF) OK. > +static int files_sync_loose_ref(struct ref_lock *lock, struct strbuf *err) This file-scope static function will not be in the vtable (in other words, it is not like "sync_loose_ref" method must be defined across all ref backends and this function is called as the implementation of the method for the files backend), so we do not have to give it "files_" prefix if we do not want to. sync_loose_ref() may be easier to read, perhaps? > +{ > + int ret = fsync_component(FSYNC_COMPONENT_LOOSE_REF, get_lock_file_fd(&lock->lk)); > + if (ret) > + strbuf_addf(err, "could not sync loose ref '%s': %s", lock->ref_name, > + strerror(errno)); > + return ret; > +} OK. Good illustration how the new helper in 6/8 is useful. It would be nice if the reroll of the base topic by Neeraj reorders the patches to introduce fsync_component() much earlier, at the same time it introduces the fsync_component_or_die(). > @@ -1504,6 +1513,7 @@ static int files_copy_or_rename_ref(struct ref_store *ref_store, > oidcpy(&lock->old_oid, &orig_oid); > > if (write_ref_to_lockfile(lock, &orig_oid, 0, &err) || > + files_sync_loose_ref(lock, &err) || > commit_ref_update(refs, lock, &orig_oid, logmsg, &err)) { > error("unable to write current sha1 into %s: %s", newrefname, err.buf); > strbuf_release(&err); > @@ -1524,6 +1534,7 @@ static int files_copy_or_rename_ref(struct ref_store *ref_store, > flag = log_all_ref_updates; > log_all_ref_updates = LOG_REFS_NONE; > if (write_ref_to_lockfile(lock, &orig_oid, 0, &err) || > + files_sync_loose_ref(lock, &err) || > commit_ref_update(refs, lock, &orig_oid, NULL, &err)) { > error("unable to write current sha1 into %s: %s", oldrefname, err.buf); > strbuf_release(&err); We used to skip commit_ref_update() and gone to the error code path upon write failure, and we do the same upon fsync failure now. The error code path will do the rollback the same way as before. OK, these look sensible. > @@ -2819,6 +2830,24 @@ static int files_transaction_prepare(struct ref_store *ref_store, > } > } > > + /* > + * Sync all lockfiles to disk to ensure data consistency. We do this in > + * a separate step such that we can sync all modified refs in a single > + * step, which may be more efficient on some filesystems. > + */ > + for (i = 0; i < transaction->nr; i++) { > + struct ref_update *update = transaction->updates[i]; > + struct ref_lock *lock = update->backend_data; > + > + if (!(update->flags & REF_NEEDS_COMMIT)) > + continue; > + > + if (files_sync_loose_ref(lock, err)) { > + ret = TRANSACTION_GENERIC_ERROR; > + goto cleanup; > + } > + } An obvious alternative that naïvely comes to mind is to keep going after seeing the first failure to sync and attempt to sync all the rest, but remember that we had an error. But I think what this patch does makes a lot more sense, as the error code path will just cleans the transaction up. If any of them fails, there is no reason to spend more effort. Thanks. > cleanup: > free(head_ref); > string_list_clear(&affected_refnames, 0); ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH 7/8] core.fsync: new option to harden loose references 2022-03-10 18:25 ` Junio C Hamano @ 2022-03-10 19:03 ` Neeraj Singh 0 siblings, 0 replies; 122+ messages in thread From: Neeraj Singh @ 2022-03-10 19:03 UTC (permalink / raw) To: Junio C Hamano Cc: Patrick Steinhardt, Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Ævar Arnfjörð Bjarmason, brian m. carlson, Neeraj K. Singh > Good illustration how the new helper in 6/8 is useful. It would be > nice if the reroll of the base topic by Neeraj reorders the patches > to introduce fsync_component() much earlier, at the same time it > introduces the fsync_component_or_die(). Will do. ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH 7/8] core.fsync: new option to harden loose references 2022-03-10 9:53 ` [PATCH 7/8] core.fsync: new option to harden loose references Patrick Steinhardt 2022-03-10 18:25 ` Junio C Hamano @ 2022-03-10 22:54 ` Neeraj Singh 2022-03-11 6:40 ` Junio C Hamano 2 siblings, 0 replies; 122+ messages in thread From: Neeraj Singh @ 2022-03-10 22:54 UTC (permalink / raw) To: Patrick Steinhardt Cc: Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Ævar Arnfjörð Bjarmason, brian m. carlson, Neeraj K. Singh, Junio C Hamano On Thu, Mar 10, 2022 at 1:53 AM Patrick Steinhardt <ps@pks.im> wrote: > diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt > index 973805e8a9..b67d3c340e 100644 > --- a/Documentation/config/core.txt > +++ b/Documentation/config/core.txt > @@ -564,8 +564,10 @@ core.fsync:: > * `pack-metadata` hardens packfile bitmaps and indexes. > * `commit-graph` hardens the commit graph file. > * `index` hardens the index when it is modified. > +* `loose-ref` hardens references modified in the repo in loose-ref form. > * `objects` is an aggregate option that is equivalent to > `loose-object,pack`. > +* `refs` is an aggregate option that is equivalent to `loose-ref`. > * `derived-metadata` is an aggregate option that is equivalent to > `pack-metadata,commit-graph`. > * `default` is an aggregate option that is equivalent to > diff --git a/cache.h b/cache.h > index 63a95d1977..b56a56f539 100644 > --- a/cache.h > +++ b/cache.h > @@ -1005,11 +1005,14 @@ enum fsync_component { > FSYNC_COMPONENT_PACK_METADATA = 1 << 2, > FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3, > FSYNC_COMPONENT_INDEX = 1 << 4, > + FSYNC_COMPONENT_LOOSE_REF = 1 << 5, > }; > > #define FSYNC_COMPONENTS_OBJECTS (FSYNC_COMPONENT_LOOSE_OBJECT | \ > FSYNC_COMPONENT_PACK) > > +#define FSYNC_COMPONENTS_REFS (FSYNC_COMPONENT_LOOSE_REF) > + > #define FSYNC_COMPONENTS_DERIVED_METADATA (FSYNC_COMPONENT_PACK_METADATA | \ > FSYNC_COMPONENT_COMMIT_GRAPH) > > @@ -1026,7 +1029,8 @@ enum fsync_component { > FSYNC_COMPONENT_PACK | \ > FSYNC_COMPONENT_PACK_METADATA | \ > FSYNC_COMPONENT_COMMIT_GRAPH | \ > - FSYNC_COMPONENT_INDEX) > + FSYNC_COMPONENT_INDEX | \ > + FSYNC_COMPONENT_LOOSE_REF) > > /* > * A bitmask indicating which components of the repo should be fsynced. > diff --git a/config.c b/config.c > index f03f27c3de..b5d3e6e404 100644 > --- a/config.c > +++ b/config.c > @@ -1332,7 +1332,9 @@ static const struct fsync_component_entry { > { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA }, > { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, > { "index", FSYNC_COMPONENT_INDEX }, > + { "loose-ref", FSYNC_COMPONENT_LOOSE_REF }, > { "objects", FSYNC_COMPONENTS_OBJECTS }, > + { "refs", FSYNC_COMPONENTS_REFS }, > { "derived-metadata", FSYNC_COMPONENTS_DERIVED_METADATA }, > { "default", FSYNC_COMPONENTS_DEFAULT }, > { "committed", FSYNC_COMPONENTS_COMMITTED }, In terms of the 'preciousness-levels', refs should be included in FSYNC_COMPONENTS_COMMITTED, from which it will also be included in _ADDED. ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH 7/8] core.fsync: new option to harden loose references 2022-03-10 9:53 ` [PATCH 7/8] core.fsync: new option to harden loose references Patrick Steinhardt 2022-03-10 18:25 ` Junio C Hamano 2022-03-10 22:54 ` Neeraj Singh @ 2022-03-11 6:40 ` Junio C Hamano 2022-03-11 9:15 ` Patrick Steinhardt 2 siblings, 1 reply; 122+ messages in thread From: Junio C Hamano @ 2022-03-11 6:40 UTC (permalink / raw) To: Patrick Steinhardt Cc: git, rsbecker, bagasdotme, newren, avarab, nksingh85, sandals, Neeraj K. Singh Patrick Steinhardt <ps@pks.im> writes: > @@ -1504,6 +1513,7 @@ static int files_copy_or_rename_ref(struct ref_store *ref_store, > oidcpy(&lock->old_oid, &orig_oid); > > if (write_ref_to_lockfile(lock, &orig_oid, 0, &err) || > + files_sync_loose_ref(lock, &err) || > commit_ref_update(refs, lock, &orig_oid, logmsg, &err)) { > error("unable to write current sha1 into %s: %s", newrefname, err.buf); > strbuf_release(&err); Given that write_ref_to_lockfile() on the success code path does this: fd = get_lock_file_fd(&lock->lk); if (write_in_full(fd, oid_to_hex(oid), the_hash_algo->hexsz) < 0 || write_in_full(fd, &term, 1) < 0 || close_ref_gently(lock) < 0) { strbuf_addf(err, "couldn't write '%s'", get_lock_file_path(&lock->lk)); unlock_ref(lock); return -1; } return 0; the above unfortunately does not work. By the time the new call to files_sync_loose_ref() is made, lock->fd is closed by the call to close_lock_file_gently() made in close_ref_gently(), and because of that, you'll get an error like this: Writing objects: 100% (3/3), 279 bytes | 279.00 KiB/s, done. Total 3 (delta 0), reused 0 (delta 0), pack-reused 0 remote: error: could not sync loose ref 'refs/heads/client_branch': Bad file descriptor when running "make test" (the above is from t5702 but I wouldn't be surprised if this broke ALL ref updates). Just before write_ref_to_lockfile() calls close_ref_gently() would be a good place to make the fsync_loose_ref() call, perhaps? Thanks. ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH 7/8] core.fsync: new option to harden loose references 2022-03-11 6:40 ` Junio C Hamano @ 2022-03-11 9:15 ` Patrick Steinhardt 2022-03-11 9:36 ` Ævar Arnfjörð Bjarmason 0 siblings, 1 reply; 122+ messages in thread From: Patrick Steinhardt @ 2022-03-11 9:15 UTC (permalink / raw) To: Junio C Hamano Cc: git, rsbecker, bagasdotme, newren, avarab, nksingh85, sandals, Neeraj K. Singh [-- Attachment #1: Type: text/plain, Size: 2202 bytes --] On Thu, Mar 10, 2022 at 10:40:07PM -0800, Junio C Hamano wrote: > Patrick Steinhardt <ps@pks.im> writes: > > > @@ -1504,6 +1513,7 @@ static int files_copy_or_rename_ref(struct ref_store *ref_store, > > oidcpy(&lock->old_oid, &orig_oid); > > > > if (write_ref_to_lockfile(lock, &orig_oid, 0, &err) || > > + files_sync_loose_ref(lock, &err) || > > commit_ref_update(refs, lock, &orig_oid, logmsg, &err)) { > > error("unable to write current sha1 into %s: %s", newrefname, err.buf); > > strbuf_release(&err); > > Given that write_ref_to_lockfile() on the success code path does this: > > fd = get_lock_file_fd(&lock->lk); > if (write_in_full(fd, oid_to_hex(oid), the_hash_algo->hexsz) < 0 || > write_in_full(fd, &term, 1) < 0 || > close_ref_gently(lock) < 0) { > strbuf_addf(err, > "couldn't write '%s'", get_lock_file_path(&lock->lk)); > unlock_ref(lock); > return -1; > } > return 0; > > the above unfortunately does not work. By the time the new call to > files_sync_loose_ref() is made, lock->fd is closed by the call to > close_lock_file_gently() made in close_ref_gently(), and because of > that, you'll get an error like this: > > Writing objects: 100% (3/3), 279 bytes | 279.00 KiB/s, done. > Total 3 (delta 0), reused 0 (delta 0), pack-reused 0 > remote: error: could not sync loose ref 'refs/heads/client_branch': > Bad file descriptor > > when running "make test" (the above is from t5702 but I wouldn't be > surprised if this broke ALL ref updates). > > Just before write_ref_to_lockfile() calls close_ref_gently() would > be a good place to make the fsync_loose_ref() call, perhaps? > > > Thanks. Yeah, that thought indeed occurred to me this night, too. I was hoping that I could fix this before anybody noticed ;) It's a bit unfortunate that we can't just defer this to a later place to hopefully implement this more efficiently, but so be it. The alternative would be to re-open all locked loose refs and then sync them to disk, but this would likely be a lot more painful than just syncing them to disk before closing it. Will fix, thanks. Patrick [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH 7/8] core.fsync: new option to harden loose references 2022-03-11 9:15 ` Patrick Steinhardt @ 2022-03-11 9:36 ` Ævar Arnfjörð Bjarmason 0 siblings, 0 replies; 122+ messages in thread From: Ævar Arnfjörð Bjarmason @ 2022-03-11 9:36 UTC (permalink / raw) To: Patrick Steinhardt Cc: Junio C Hamano, git, rsbecker, bagasdotme, newren, nksingh85, sandals, Neeraj K. Singh On Fri, Mar 11 2022, Patrick Steinhardt wrote: > [[PGP Signed Part:Undecided]] > On Thu, Mar 10, 2022 at 10:40:07PM -0800, Junio C Hamano wrote: >> Patrick Steinhardt <ps@pks.im> writes: >> >> > @@ -1504,6 +1513,7 @@ static int files_copy_or_rename_ref(struct ref_store *ref_store, >> > oidcpy(&lock->old_oid, &orig_oid); >> > >> > if (write_ref_to_lockfile(lock, &orig_oid, 0, &err) || >> > + files_sync_loose_ref(lock, &err) || >> > commit_ref_update(refs, lock, &orig_oid, logmsg, &err)) { >> > error("unable to write current sha1 into %s: %s", newrefname, err.buf); >> > strbuf_release(&err); >> >> Given that write_ref_to_lockfile() on the success code path does this: >> >> fd = get_lock_file_fd(&lock->lk); >> if (write_in_full(fd, oid_to_hex(oid), the_hash_algo->hexsz) < 0 || >> write_in_full(fd, &term, 1) < 0 || >> close_ref_gently(lock) < 0) { >> strbuf_addf(err, >> "couldn't write '%s'", get_lock_file_path(&lock->lk)); >> unlock_ref(lock); >> return -1; >> } >> return 0; >> >> the above unfortunately does not work. By the time the new call to >> files_sync_loose_ref() is made, lock->fd is closed by the call to >> close_lock_file_gently() made in close_ref_gently(), and because of >> that, you'll get an error like this: >> >> Writing objects: 100% (3/3), 279 bytes | 279.00 KiB/s, done. >> Total 3 (delta 0), reused 0 (delta 0), pack-reused 0 >> remote: error: could not sync loose ref 'refs/heads/client_branch': >> Bad file descriptor >> >> when running "make test" (the above is from t5702 but I wouldn't be >> surprised if this broke ALL ref updates). >> >> Just before write_ref_to_lockfile() calls close_ref_gently() would >> be a good place to make the fsync_loose_ref() call, perhaps? >> >> >> Thanks. > > Yeah, that thought indeed occurred to me this night, too. I was hoping > that I could fix this before anybody noticed ;) > > It's a bit unfortunate that we can't just defer this to a later place to > hopefully implement this more efficiently, but so be it. The alternative > would be to re-open all locked loose refs and then sync them to disk, > but this would likely be a lot more painful than just syncing them to > disk before closing it. Aside: is open/write/close followed by open/fsync/close on the same file portably guaranteed to yield the same end result as a single open/write/fsync/close? I think in practice nobody would be insane enough to implement a system to do otherwise, but on the other hand I've seen some really insane behavior :) I could see it being different e.g. in some NFS cases/configurations where the fsync() for an open FD syncs to the remote storage, and the second open() might therefore get the old version and noop-sync that. Most implementations would guard against that in the common case by having a local cache of outstanding data to flush, but if you're talking to some sharded storage array for each request... Anyway, I *think* it should be OK, just an aside to check the assumption for any future work... :) ^ permalink raw reply [flat|nested] 122+ messages in thread
* [PATCH 8/8] core.fsync: new option to harden packed references 2022-03-09 23:03 ` [PATCH v5 0/5] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget ` (7 preceding siblings ...) 2022-03-10 9:53 ` [PATCH 7/8] core.fsync: new option to harden loose references Patrick Steinhardt @ 2022-03-10 9:53 ` Patrick Steinhardt 2022-03-10 18:28 ` Junio C Hamano 2022-03-10 22:43 ` [PATCH v6 0/6] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget 9 siblings, 1 reply; 122+ messages in thread From: Patrick Steinhardt @ 2022-03-10 9:53 UTC (permalink / raw) To: git Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, sandals, Neeraj K. Singh, Junio C Hamano [-- Attachment #1: Type: text/plain, Size: 3485 bytes --] Similar to the preceding commit, this commit adds a new option to harden packed references so that users and admins can avoid data loss when we commit a new packed-refs file. Signed-off-by: Patrick Steinhardt <ps@pks.im> --- Documentation/config/core.txt | 3 ++- cache.h | 7 +++++-- config.c | 1 + refs/packed-backend.c | 3 ++- 4 files changed, 10 insertions(+), 4 deletions(-) diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt index b67d3c340e..3fd466f955 100644 --- a/Documentation/config/core.txt +++ b/Documentation/config/core.txt @@ -565,9 +565,10 @@ core.fsync:: * `commit-graph` hardens the commit graph file. * `index` hardens the index when it is modified. * `loose-ref` hardens references modified in the repo in loose-ref form. +* `packed-refs` hardens references modified in the repo in packed-refs form. * `objects` is an aggregate option that is equivalent to `loose-object,pack`. -* `refs` is an aggregate option that is equivalent to `loose-ref`. +* `refs` is an aggregate option that is equivalent to `loose-ref,packed-refs`. * `derived-metadata` is an aggregate option that is equivalent to `pack-metadata,commit-graph`. * `default` is an aggregate option that is equivalent to diff --git a/cache.h b/cache.h index b56a56f539..9b7c282fa5 100644 --- a/cache.h +++ b/cache.h @@ -1006,12 +1006,14 @@ enum fsync_component { FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3, FSYNC_COMPONENT_INDEX = 1 << 4, FSYNC_COMPONENT_LOOSE_REF = 1 << 5, + FSYNC_COMPONENT_PACKED_REFS = 1 << 6, }; #define FSYNC_COMPONENTS_OBJECTS (FSYNC_COMPONENT_LOOSE_OBJECT | \ FSYNC_COMPONENT_PACK) -#define FSYNC_COMPONENTS_REFS (FSYNC_COMPONENT_LOOSE_REF) +#define FSYNC_COMPONENTS_REFS (FSYNC_COMPONENT_LOOSE_REF | \ + FSYNC_COMPONENT_PACKED_REFS) #define FSYNC_COMPONENTS_DERIVED_METADATA (FSYNC_COMPONENT_PACK_METADATA | \ FSYNC_COMPONENT_COMMIT_GRAPH) @@ -1030,7 +1032,8 @@ enum fsync_component { FSYNC_COMPONENT_PACK_METADATA | \ FSYNC_COMPONENT_COMMIT_GRAPH | \ FSYNC_COMPONENT_INDEX | \ - FSYNC_COMPONENT_LOOSE_REF) + FSYNC_COMPONENT_LOOSE_REF | \ + FSYNC_COMPONENT_PACKED_REFS) /* * A bitmask indicating which components of the repo should be fsynced. diff --git a/config.c b/config.c index b5d3e6e404..b4a2ee3a8c 100644 --- a/config.c +++ b/config.c @@ -1333,6 +1333,7 @@ static const struct fsync_component_entry { { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, { "index", FSYNC_COMPONENT_INDEX }, { "loose-ref", FSYNC_COMPONENT_LOOSE_REF }, + { "packed-refs", FSYNC_COMPONENT_PACKED_REFS }, { "objects", FSYNC_COMPONENTS_OBJECTS }, { "refs", FSYNC_COMPONENTS_REFS }, { "derived-metadata", FSYNC_COMPONENTS_DERIVED_METADATA }, diff --git a/refs/packed-backend.c b/refs/packed-backend.c index 27dd8c3922..32d6635969 100644 --- a/refs/packed-backend.c +++ b/refs/packed-backend.c @@ -1262,7 +1262,8 @@ static int write_with_updates(struct packed_ref_store *refs, goto error; } - if (close_tempfile_gently(refs->tempfile)) { + if (fsync_component(FSYNC_COMPONENT_PACKED_REFS, get_tempfile_fd(refs->tempfile)) || + close_tempfile_gently(refs->tempfile)) { strbuf_addf(err, "error closing file %s: %s", get_tempfile_path(refs->tempfile), strerror(errno)); -- 2.35.1 [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply related [flat|nested] 122+ messages in thread
* Re: [PATCH 8/8] core.fsync: new option to harden packed references 2022-03-10 9:53 ` [PATCH 8/8] core.fsync: new option to harden packed references Patrick Steinhardt @ 2022-03-10 18:28 ` Junio C Hamano 2022-03-11 9:10 ` Patrick Steinhardt 0 siblings, 1 reply; 122+ messages in thread From: Junio C Hamano @ 2022-03-10 18:28 UTC (permalink / raw) To: Patrick Steinhardt Cc: git, rsbecker, bagasdotme, newren, avarab, nksingh85, sandals, Neeraj K. Singh Patrick Steinhardt <ps@pks.im> writes: > diff --git a/refs/packed-backend.c b/refs/packed-backend.c > index 27dd8c3922..32d6635969 100644 > --- a/refs/packed-backend.c > +++ b/refs/packed-backend.c > @@ -1262,7 +1262,8 @@ static int write_with_updates(struct packed_ref_store *refs, > goto error; > } > > - if (close_tempfile_gently(refs->tempfile)) { > + if (fsync_component(FSYNC_COMPONENT_PACKED_REFS, get_tempfile_fd(refs->tempfile)) || > + close_tempfile_gently(refs->tempfile)) { > strbuf_addf(err, "error closing file %s: %s", > get_tempfile_path(refs->tempfile), > strerror(errno)); I do not necessarily agree with the organization to have it as a component that is separate from other ref backends, but it is very pleasing to see that there is only one fsync necessary for the packed backend. Nice. ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH 8/8] core.fsync: new option to harden packed references 2022-03-10 18:28 ` Junio C Hamano @ 2022-03-11 9:10 ` Patrick Steinhardt 0 siblings, 0 replies; 122+ messages in thread From: Patrick Steinhardt @ 2022-03-11 9:10 UTC (permalink / raw) To: Junio C Hamano Cc: git, rsbecker, bagasdotme, newren, avarab, nksingh85, sandals, Neeraj K. Singh [-- Attachment #1: Type: text/plain, Size: 1313 bytes --] On Thu, Mar 10, 2022 at 10:28:57AM -0800, Junio C Hamano wrote: > Patrick Steinhardt <ps@pks.im> writes: > > > diff --git a/refs/packed-backend.c b/refs/packed-backend.c > > index 27dd8c3922..32d6635969 100644 > > --- a/refs/packed-backend.c > > +++ b/refs/packed-backend.c > > @@ -1262,7 +1262,8 @@ static int write_with_updates(struct packed_ref_store *refs, > > goto error; > > } > > > > - if (close_tempfile_gently(refs->tempfile)) { > > + if (fsync_component(FSYNC_COMPONENT_PACKED_REFS, get_tempfile_fd(refs->tempfile)) || > > + close_tempfile_gently(refs->tempfile)) { > > strbuf_addf(err, "error closing file %s: %s", > > get_tempfile_path(refs->tempfile), > > strerror(errno)); > > I do not necessarily agree with the organization to have it as a > component that is separate from other ref backends, but it is > very pleasing to see that there is only one fsync necessary for the > packed backend. > > Nice. I was mostly adapting to the precedent set by Neeraj, where we also distinguish loose objects and packed objects. Personally I don't mind much whether we want to discern those two cases, and I'd be happy to just merge them into a single "refs" knob. We can still split these up at a later point if the need ever arises. Patrick [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 122+ messages in thread
* [PATCH v6 0/6] A design for future-proofing fsync() configuration 2022-03-09 23:03 ` [PATCH v5 0/5] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget ` (8 preceding siblings ...) 2022-03-10 9:53 ` [PATCH 8/8] core.fsync: new option to harden packed references Patrick Steinhardt @ 2022-03-10 22:43 ` Neeraj K. Singh via GitGitGadget 2022-03-10 22:43 ` [PATCH v6 1/6] wrapper: make inclusion of Windows csprng header tightly scoped Neeraj Singh via GitGitGadget ` (7 more replies) 9 siblings, 8 replies; 122+ messages in thread From: Neeraj K. Singh via GitGitGadget @ 2022-03-10 22:43 UTC (permalink / raw) To: git Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps, sandals, Johannes Schindelin, Neeraj K. Singh This is an implementation of an extensible configuration mechanism for fsyncing persistent components of a repo. The main goals are to separate the "what" to sync from the "how". There are now two settings: core.fsync - Control the 'what', including the index. core.fsyncMethod - Control the 'how'. Currently we support writeout-only and full fsync. Syncing of refs can be layered on top of core.fsync. And batch mode will be layered on core.fsyncMethod. Once this series reaches 'seen', I'll submit ns/batched-fsync to introduce batch mode. Please see https://github.com/gitgitgadget/git/pull/1134. core.fsyncObjectfiles is removed and we will issue a deprecation warning if it's seen. I'd like to get agreement on this direction before submitting batch mode to the list. The batch mode series is available to view at Please see [1], [2], and [3] for discussions that led to this series. After this change, new persistent data files added to the repo will need to be added to the fsync_component enum and documented in the Documentation/config/core.txt text. V6 changes: * Move only the windows csprng includes into wrapper.c rather than all of them. This fixes the specific build issue due to broken Windows headers. [6] * Split the configuration parsing of core.fsync from the mechanism to focus the review. * Incorporate Patrick's patch at [7] into the core.fsync mechanism patch. * Pick the stricter one of core.fsyncObjectFiles and (fsync_components & FSYNC_COMPONENT_LOOSE_OBJECTS), to respect the older setting. * Issue a deprecation warning but keep parsing and honoring core.fsyncObjectFiles. * Change configuration parsing of core.fsync to always start with the platform default. none resets to the empty set. The comma separated list implies a set without regards to ordering now. This follows Junio's suggestion in [8]. * Change the documentation of the core.fsync option to reflect the way the new parsing code works. * The patch 7 and 8 of Patrick's series at [7] can be cherry-picked after being applied to ns/core-fsyncmethod. V5 changes: * Rebase onto main at c2162907e9 * Add a patch to move CSPRNG platform includes to wrapper.c. This avoids build errors in compat/win32/flush.c and other files. * Move the documentation and aggregate options to the final patch in the series. * Define new aggregate options and guidance in line with Junio's suggestion to present the user with 'levels of safety' rather than a morass of detailed options. V4 changes: * Rebase onto master at b23dac905bd. * Add a comment to write_pack_file indicating why we don't fsync when writing to stdout. * I kept the configuration schema as-is rather than switching to multi-value. The thinking here is that a stateless last-one-wins config schema (comma separated) will make it easier to achieve some holistic self-consistent fsync configuration for a particular repo. V3 changes: * Remove relative path from git-compat-util.h include [4]. * Updated newly added warning texts to have more context for localization [4]. * Fixed tab spacing in enum fsync_action * Moved the fsync looping out to a helper and do it consistently. [4] * Changed commit description to use camelCase for config names. [5] * Add an optional fourth patch with derived-metadata so that the user can exclude a forward-compatible set of things that should be recomputable given existing data. V2 changes: * Updated the documentation for core.fsyncmethod to be less certain. writeout-only probably does not do the right thing on Linux. * Split out the core.fsync=index change into its own commit. * Rename REPO_COMPONENT to FSYNC_COMPONENT. This is really specific to fsyncing, so the name should reflect that. * Re-add missing Makefile change for SYNC_FILE_RANGE. * Tested writeout-only mode, index syncing, and general config settings. [1] https://lore.kernel.org/git/211110.86r1bogg27.gmgdl@evledraar.gmail.com/ [2] https://lore.kernel.org/git/dd65718814011eb93ccc4428f9882e0f025224a6.1636029491.git.ps@pks.im/ [3] https://lore.kernel.org/git/pull.1076.git.git.1629856292.gitgitgadget@gmail.com/ [4] https://lore.kernel.org/git/CANQDOdf8C4-haK9=Q_J4Cid8bQALnmGDm=SvatRbaVf+tkzqLw@mail.gmail.com/ [5] https://lore.kernel.org/git/211207.861r2opplg.gmgdl@evledraar.gmail.com/ [6] https://lore.kernel.org/git/CANQDOdfZbOHZQt9Ah0t1AamTO2T7Gq0tmWX1jLqL6njE0LF6DA@mail.gmail.com/ [7] https://lore.kernel.org/git/50e39f698a7c0cc06d3bc060e6dbc539ea693241.1646905589.git.ps@pks.im/ [8] https://lore.kernel.org/git/xmqqk0d1cxsv.fsf@gitster.g/ Neeraj Singh (6): wrapper: make inclusion of Windows csprng header tightly scoped core.fsyncmethod: add writeout-only mode core.fsync: introduce granular fsync control infrastructure core.fsync: add configuration parsing core.fsync: new option to harden the index core.fsync: documentation and user-friendly aggregate options Documentation/config/core.txt | 58 ++++++++++++++++-- Makefile | 6 ++ builtin/fast-import.c | 2 +- builtin/index-pack.c | 4 +- builtin/pack-objects.c | 24 +++++--- bulk-checkin.c | 5 +- cache.h | 48 +++++++++++++++ commit-graph.c | 3 +- compat/mingw.h | 3 + compat/win32/flush.c | 28 +++++++++ compat/winansi.c | 5 -- config.c | 94 +++++++++++++++++++++++++++++ config.mak.uname | 3 + configure.ac | 8 +++ contrib/buildsystems/CMakeLists.txt | 16 +++-- csum-file.c | 5 +- csum-file.h | 3 +- environment.c | 4 +- git-compat-util.h | 30 +++++++-- midx.c | 3 +- object-file.c | 13 ++-- pack-bitmap-write.c | 3 +- pack-write.c | 13 ++-- read-cache.c | 19 ++++-- wrapper.c | 71 ++++++++++++++++++++++ write-or-die.c | 33 ++++++++-- 26 files changed, 444 insertions(+), 60 deletions(-) create mode 100644 compat/win32/flush.c base-commit: c2162907e9aa884bdb70208389cb99b181620d51 Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1093%2Fneerajsi-msft%2Fns%2Fcore-fsync-v6 Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1093/neerajsi-msft/ns/core-fsync-v6 Pull-Request: https://github.com/gitgitgadget/git/pull/1093 Range-diff vs v5: 1: 685b1db8880 ! 1: 825079b6aa1 wrapper: move inclusion of CSPRNG headers the wrapper.c file @@ Metadata Author: Neeraj Singh <neerajsi@microsoft.com> ## Commit message ## - wrapper: move inclusion of CSPRNG headers the wrapper.c file + wrapper: make inclusion of Windows csprng header tightly scoped Including NTSecAPI.h in git-compat-util.h causes build errors in any - other file that includes winternl.h. That file was included in order to + other file that includes winternl.h. NTSecAPI.h was included in order to get access to the RtlGenRandom cryptographically secure PRNG. This - change scopes the inclusion of all PRNG headers to just the wrapper.c - file, which is the only place it is really needed. + change scopes the inclusion of ntsecapi.h to wrapper.c, which is the only + place that it's actually needed. + + The build breakage is due to the definition of UNICODE_STRING in + NtSecApi.h: + #ifndef _NTDEF_ + typedef LSA_UNICODE_STRING UNICODE_STRING, *PUNICODE_STRING; + typedef LSA_STRING STRING, *PSTRING ; + #endif + + LsaLookup.h: + typedef struct _LSA_UNICODE_STRING { + USHORT Length; + USHORT MaximumLength; + #ifdef MIDL_PASS + [size_is(MaximumLength/2), length_is(Length/2)] + #endif // MIDL_PASS + PWSTR Buffer; + } LSA_UNICODE_STRING, *PLSA_UNICODE_STRING; + + winternl.h also defines UNICODE_STRING: + typedef struct _UNICODE_STRING { + USHORT Length; + USHORT MaximumLength; + PWSTR Buffer; + } UNICODE_STRING; + typedef UNICODE_STRING *PUNICODE_STRING; + + Both definitions have equivalent layouts. Apparently these internal + Windows headers aren't designed to be included together. This is + an oversight in the headers and does not represent an incompatibility + between the APIs. Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> @@ git-compat-util.h #endif #include <unistd.h> -@@ - #else - #include <stdint.h> - #endif --#ifdef HAVE_ARC4RANDOM_LIBBSD --#include <bsd/stdlib.h> --#endif --#ifdef HAVE_GETRANDOM --#include <sys/random.h> --#endif - #ifdef NO_INTPTR_T - /* - * On I16LP32, ILP32 and LP64 "long" is the safe bet, however ## wrapper.c ## @@ @@ wrapper.c +#include <NTSecAPI.h> +#undef SystemFunction036 +#endif -+ -+#ifdef HAVE_ARC4RANDOM_LIBBSD -+#include <bsd/stdlib.h> -+#endif -+#ifdef HAVE_GETRANDOM -+#include <sys/random.h> -+#endif + static int memory_limit_check(size_t size, int gentle) { 2: da8cfc10bb4 = 2: a41bd4a06af core.fsyncmethod: add writeout-only mode 3: e31886717b4 ! 3: 64e2bdcdfd9 core.fsync: introduce granular fsync control @@ Metadata Author: Neeraj Singh <neerajsi@microsoft.com> ## Commit message ## - core.fsync: introduce granular fsync control + core.fsync: introduce granular fsync control infrastructure - This commit introduces the `core.fsync` configuration - knob which can be used to control how components of the - repository are made durable on disk. + This commit introduces the infrastructure for the core.fsync + configuration knob. The repository components we want to sync + are identified by flags so that we can turn on or off syncing + for specific components. - This setting allows future extensibility of the list of - syncable components: - * We issue a warning rather than an error for unrecognized - components, so new configs can be used with old Git versions. - * We support negation, so users can choose one of the aggregate - options and then remove components that they don't want. - Aggregate options are defined in a later patch in this series. + If core.fsyncObjectFiles is set and the core.fsync configuration + also includes FSYNC_COMPONENT_LOOSE_OBJECT, we will fsync any + loose objects. This picks the strictest data integrity behavior + if core.fsync and core.fsyncObjectFiles are set to conflicting values. - This also supports the common request of doing absolutely no - fysncing with the `core.fsync=none` value, which is expected - to make the test suite faster. + This change introduces the currently unused fsync_component + helper, which will be used by a later patch that adds fsyncing to + the refs backend. - Complete documentation for the new setting is included in a later patch - in the series so that it can be reviewed in final form. + Actual configuration and documentation of the fsync components + list are in other patches in the series to separate review of + the underlying mechanism from the policy of how it's configured. + Helped-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> - ## Documentation/config/core.txt ## -@@ Documentation/config/core.txt: core.fsyncMethod:: - filesystem and storage hardware, data added to the repository may not be - durable in the event of a system crash. This is the default mode on macOS. - --core.fsyncObjectFiles:: -- This boolean will enable 'fsync()' when writing object files. --+ --This is a total waste of time and effort on a filesystem that orders --data writes properly, but can be useful for filesystems that do not use --journalling (traditional UNIX filesystems) or that only journal metadata --and not file contents (OS X's HFS+, or Linux ext3 with "data=writeback"). -- - core.preloadIndex:: - Enable parallel index preload for operations like 'git diff' - + - ## builtin/fast-import.c ## @@ builtin/fast-import.c: static void end_packfile(void) struct tag *t; @@ cache.h: void reset_shared_repository(void); extern int read_replace_refs; extern char *git_replace_ref_base; --extern int fsync_object_files; --extern int use_fsync; +/* + * These values are used to help identify parts of a repository to fsync. + * FSYNC_COMPONENT_NONE identifies data that will not be a persistent part of the @@ cache.h: void reset_shared_repository(void); + * A bitmask indicating which components of the repo should be fsynced. + */ +extern enum fsync_component fsync_components; + extern int fsync_object_files; + extern int use_fsync; - enum fsync_method { - FSYNC_METHOD_FSYNC, -@@ cache.h: enum fsync_method { - }; - - extern enum fsync_method fsync_method; -+extern int use_fsync; - extern int core_preload_index; - extern int precomposed_unicode; - extern int protect_hfs; @@ cache.h: int copy_file_with_time(const char *dst, const char *src, int mode); + void write_or_die(int fd, const void *buf, size_t count); void fsync_or_die(int fd, const char *); ++int fsync_component(enum fsync_component component, int fd); ++void fsync_component_or_die(enum fsync_component component, int fd, const char *msg); -+static inline void fsync_component_or_die(enum fsync_component component, int fd, const char *msg) -+{ -+ if (fsync_components & component) -+ fsync_or_die(fd, msg); -+} -+ ssize_t read_in_full(int fd, void *buf, size_t count); ssize_t write_in_full(int fd, const void *buf, size_t count); - ssize_t pread_in_full(int fd, void *buf, size_t count, off_t offset); ## commit-graph.c ## @@ commit-graph.c: static int write_commit_graph_file(struct write_commit_graph_context *ctx) @@ commit-graph.c: static int write_commit_graph_file(struct write_commit_graph_con if (ctx->split) { - ## config.c ## -@@ config.c: static int git_parse_maybe_bool_text(const char *value) - return -1; - } - -+static const struct fsync_component_entry { -+ const char *name; -+ enum fsync_component component_bits; -+} fsync_component_table[] = { -+ { "loose-object", FSYNC_COMPONENT_LOOSE_OBJECT }, -+ { "pack", FSYNC_COMPONENT_PACK }, -+ { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA }, -+ { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, -+}; -+ -+static enum fsync_component parse_fsync_components(const char *var, const char *string) -+{ -+ enum fsync_component output = 0; -+ -+ if (!strcmp(string, "none")) -+ return FSYNC_COMPONENT_NONE; -+ -+ while (string) { -+ int i; -+ size_t len; -+ const char *ep; -+ int negated = 0; -+ int found = 0; -+ -+ string = string + strspn(string, ", \t\n\r"); -+ ep = strchrnul(string, ','); -+ len = ep - string; -+ -+ if (*string == '-') { -+ negated = 1; -+ string++; -+ len--; -+ if (!len) -+ warning(_("invalid value for variable %s"), var); -+ } -+ -+ if (!len) -+ break; -+ -+ for (i = 0; i < ARRAY_SIZE(fsync_component_table); ++i) { -+ const struct fsync_component_entry *entry = &fsync_component_table[i]; -+ -+ if (strncmp(entry->name, string, len)) -+ continue; -+ -+ found = 1; -+ if (negated) -+ output &= ~entry->component_bits; -+ else -+ output |= entry->component_bits; -+ } -+ -+ if (!found) { -+ char *component = xstrndup(string, len); -+ warning(_("ignoring unknown core.fsync component '%s'"), component); -+ free(component); -+ } -+ -+ string = ep; -+ } -+ -+ return output; -+} -+ - int git_parse_maybe_bool(const char *value) - { - int v = git_parse_maybe_bool_text(value); -@@ config.c: static int git_default_core_config(const char *var, const char *value, void *cb) - return 0; - } - -+ if (!strcmp(var, "core.fsync")) { -+ if (!value) -+ return config_error_nonbool(var); -+ fsync_components = parse_fsync_components(var, value); -+ return 0; -+ } -+ - if (!strcmp(var, "core.fsyncmethod")) { - if (!value) - return config_error_nonbool(var); -@@ config.c: static int git_default_core_config(const char *var, const char *value, void *cb) - } - - if (!strcmp(var, "core.fsyncobjectfiles")) { -- fsync_object_files = git_config_bool(var, value); -+ warning(_("core.fsyncobjectfiles is deprecated; use core.fsync instead")); - return 0; - } - - ## csum-file.c ## @@ csum-file.c: static void free_hashfile(struct hashfile *f) free(f); @@ csum-file.h: int hashfile_truncate(struct hashfile *, struct hashfile_checkpoint void crc32_begin(struct hashfile *); ## environment.c ## -@@ environment.c: const char *git_attributes_file; - const char *git_hooks_path; - int zlib_compression_level = Z_BEST_SPEED; - int pack_compression_level = Z_DEFAULT_COMPRESSION; --int fsync_object_files; +@@ environment.c: int pack_compression_level = Z_DEFAULT_COMPRESSION; + int fsync_object_files; int use_fsync = -1; enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT; +enum fsync_component fsync_components = FSYNC_COMPONENTS_DEFAULT; @@ midx.c: static int write_midx_internal(const char *object_dir, ## object-file.c ## @@ object-file.c: int hash_object_file(const struct git_hash_algo *algo, const void *buf, + /* Finalize a file on disk, and close it. */ static void close_loose_object(int fd) { - if (!the_repository->objects->odb->will_destroy) { +- if (!the_repository->objects->odb->will_destroy) { - if (fsync_object_files) - fsync_or_die(fd, "loose object file"); -+ fsync_component_or_die(FSYNC_COMPONENT_LOOSE_OBJECT, fd, "loose object file"); - } +- } ++ if (the_repository->objects->odb->will_destroy) ++ goto out; ++ if (fsync_object_files > 0) ++ fsync_or_die(fd, "loose object file"); ++ else ++ fsync_component_or_die(FSYNC_COMPONENT_LOOSE_OBJECT, fd, ++ "loose object file"); ++ ++out: if (close(fd) != 0) + die_errno(_("error when closing loose object file")); + } ## pack-bitmap-write.c ## @@ pack-bitmap-write.c: void bitmap_writer_finish(struct pack_idx_entry **index, @@ read-cache.c: static int do_write_index(struct index_state *istate, struct tempf if (close_tempfile_gently(tempfile)) { error(_("could not close '%s'"), get_tempfile_path(tempfile)); return -1; + + ## write-or-die.c ## +@@ write-or-die.c: void fprintf_or_die(FILE *f, const char *fmt, ...) + } + } + +-void fsync_or_die(int fd, const char *msg) ++static int maybe_fsync(int fd) + { + if (use_fsync < 0) + use_fsync = git_env_bool("GIT_TEST_FSYNC", 1); + if (!use_fsync) +- return; ++ return 0; + + if (fsync_method == FSYNC_METHOD_WRITEOUT_ONLY && + git_fsync(fd, FSYNC_WRITEOUT_ONLY) >= 0) +- return; ++ return 0; ++ ++ return git_fsync(fd, FSYNC_HARDWARE_FLUSH); ++} + +- if (git_fsync(fd, FSYNC_HARDWARE_FLUSH) < 0) ++void fsync_or_die(int fd, const char *msg) ++{ ++ if (maybe_fsync(fd) < 0) + die_errno("fsync error on '%s'", msg); + } + ++int fsync_component(enum fsync_component component, int fd) ++{ ++ if (fsync_components & component) ++ return maybe_fsync(fd); ++ return 0; ++} ++ ++void fsync_component_or_die(enum fsync_component component, int fd, const char *msg) ++{ ++ if (fsync_components & component) ++ fsync_or_die(fd, msg); ++} ++ + void write_or_die(int fd, const void *buf, size_t count) + { + if (write_in_full(fd, buf, count) < 0) { -: ----------- > 4: 6adc8dc1385 core.fsync: add configuration parsing 4: 9da808ba743 ! 5: 757f6d0bbd2 core.fsync: new option to harden the index @@ cache.h: enum fsync_component { #define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \ ## config.c ## -@@ config.c: static const struct fsync_component_entry { +@@ config.c: static const struct fsync_component_name { { "pack", FSYNC_COMPONENT_PACK }, { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA }, { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, 5: 2d71346b10e ! 6: 7e4cc6e10a5 core.fsync: documentation and user-friendly aggregate options @@ Documentation/config/core.txt: core.whitespace:: errors. The default tab width is 8. Allowed values are 1 to 63. +core.fsync:: -+ A comma-separated list of parts of the repository which should be -+ hardened via the core.fsyncMethod when created or modified. You can -+ disable hardening of any component by prefixing it with a '-'. Later -+ items take precedence over earlier ones in the comma-separated list. -+ For example, `core.fsync=all,-pack-metadata` means "harden everything -+ except pack metadata." Items that are not hardened may be lost in the -+ event of an unclean system shutdown. Unless you have special -+ requirements, it is recommended that you leave this option as default -+ or pick one of `committed`, `added`, or `all`. ++ A comma-separated list of components of the repository that ++ should be hardened via the core.fsyncMethod when created or ++ modified. You can disable hardening of any component by ++ prefixing it with a '-'. Items that are not hardened may be ++ lost in the event of an unclean system shutdown. Unless you ++ have special requirements, it is recommended that you leave ++ this option empty or pick one of `committed`, `added`, ++ or `all`. ++ -+* `none` disables fsync completely. This value must be specified alone. ++When this configuration is encountered, the set of components starts with ++the platform default value, disabled components are removed, and additional ++components are added. `none` resets the state so that the platform default ++is ignored. +++ ++The empty string resets the fsync configuration to the platform ++default. The platform default on most platform is equivalent to ++`core.fsync=committed,-loose-object`, which has good performance, ++but risks losing recent work in the event of an unclean system shutdown. +++ ++* `none` clears the set of fsynced components. +* `loose-object` hardens objects added to the repo in loose-object form. +* `pack` hardens objects added to the repo in packfile form. +* `pack-metadata` hardens packfile bitmaps and indexes. @@ Documentation/config/core.txt: core.whitespace:: + `loose-object,pack`. +* `derived-metadata` is an aggregate option that is equivalent to + `pack-metadata,commit-graph`. -+* `default` is an aggregate option that is equivalent to -+ `objects,derived-metadata,-loose-object`. This mode is enabled by default. -+ It has good performance, but risks losing recent work if the system shuts -+ down uncleanly, since commits, trees, and blobs in loose-object form may be -+ lost. +* `committed` is an aggregate option that is currently equivalent to -+ `objects`. This mode sacrifices some performance to ensure that all work ++ `objects`. This mode sacrifices some performance to ensure that work + that is committed to the repository with `git commit` or similar commands -+ is preserved. ++ is hardened. +* `added` is an aggregate option that is currently equivalent to + `committed,index`. This mode sacrifices additional performance to + ensure that the results of commands like `git add` and similar operations -+ are preserved. ++ are hardened. +* `all` is an aggregate option that syncs all individual components above. + core.fsyncMethod:: @@ cache.h: enum fsync_component { * A bitmask indicating which components of the repo should be fsynced. ## config.c ## -@@ config.c: static const struct fsync_component_entry { +@@ config.c: static const struct fsync_component_name { { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA }, { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, { "index", FSYNC_COMPONENT_INDEX }, + { "objects", FSYNC_COMPONENTS_OBJECTS }, + { "derived-metadata", FSYNC_COMPONENTS_DERIVED_METADATA }, -+ { "default", FSYNC_COMPONENTS_DEFAULT }, + { "committed", FSYNC_COMPONENTS_COMMITTED }, + { "added", FSYNC_COMPONENTS_ADDED }, + { "all", FSYNC_COMPONENTS_ALL }, -- gitgitgadget ^ permalink raw reply [flat|nested] 122+ messages in thread
* [PATCH v6 1/6] wrapper: make inclusion of Windows csprng header tightly scoped 2022-03-10 22:43 ` [PATCH v6 0/6] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget @ 2022-03-10 22:43 ` Neeraj Singh via GitGitGadget 2022-03-10 22:43 ` [PATCH v6 2/6] core.fsyncmethod: add writeout-only mode Neeraj Singh via GitGitGadget ` (6 subsequent siblings) 7 siblings, 0 replies; 122+ messages in thread From: Neeraj Singh via GitGitGadget @ 2022-03-10 22:43 UTC (permalink / raw) To: git Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps, sandals, Johannes Schindelin, Neeraj K. Singh, Neeraj Singh From: Neeraj Singh <neerajsi@microsoft.com> Including NTSecAPI.h in git-compat-util.h causes build errors in any other file that includes winternl.h. NTSecAPI.h was included in order to get access to the RtlGenRandom cryptographically secure PRNG. This change scopes the inclusion of ntsecapi.h to wrapper.c, which is the only place that it's actually needed. The build breakage is due to the definition of UNICODE_STRING in NtSecApi.h: #ifndef _NTDEF_ typedef LSA_UNICODE_STRING UNICODE_STRING, *PUNICODE_STRING; typedef LSA_STRING STRING, *PSTRING ; #endif LsaLookup.h: typedef struct _LSA_UNICODE_STRING { USHORT Length; USHORT MaximumLength; #ifdef MIDL_PASS [size_is(MaximumLength/2), length_is(Length/2)] #endif // MIDL_PASS PWSTR Buffer; } LSA_UNICODE_STRING, *PLSA_UNICODE_STRING; winternl.h also defines UNICODE_STRING: typedef struct _UNICODE_STRING { USHORT Length; USHORT MaximumLength; PWSTR Buffer; } UNICODE_STRING; typedef UNICODE_STRING *PUNICODE_STRING; Both definitions have equivalent layouts. Apparently these internal Windows headers aren't designed to be included together. This is an oversight in the headers and does not represent an incompatibility between the APIs. Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> --- compat/winansi.c | 5 ----- git-compat-util.h | 6 ------ wrapper.c | 7 +++++++ 3 files changed, 7 insertions(+), 11 deletions(-) diff --git a/compat/winansi.c b/compat/winansi.c index 936a80a5f00..3abe8dd5a27 100644 --- a/compat/winansi.c +++ b/compat/winansi.c @@ -4,11 +4,6 @@ #undef NOGDI -/* - * Including the appropriate header file for RtlGenRandom causes MSVC to see a - * redefinition of types in an incompatible way when including headers below. - */ -#undef HAVE_RTLGENRANDOM #include "../git-compat-util.h" #include <wingdi.h> #include <winreg.h> diff --git a/git-compat-util.h b/git-compat-util.h index 876907b9df4..d210cff058c 100644 --- a/git-compat-util.h +++ b/git-compat-util.h @@ -197,12 +197,6 @@ #endif #include <windows.h> #define GIT_WINDOWS_NATIVE -#ifdef HAVE_RTLGENRANDOM -/* This is required to get access to RtlGenRandom. */ -#define SystemFunction036 NTAPI SystemFunction036 -#include <NTSecAPI.h> -#undef SystemFunction036 -#endif #endif #include <unistd.h> diff --git a/wrapper.c b/wrapper.c index 3258cdb171f..1108e4840a4 100644 --- a/wrapper.c +++ b/wrapper.c @@ -4,6 +4,13 @@ #include "cache.h" #include "config.h" +#ifdef HAVE_RTLGENRANDOM +/* This is required to get access to RtlGenRandom. */ +#define SystemFunction036 NTAPI SystemFunction036 +#include <NTSecAPI.h> +#undef SystemFunction036 +#endif + static int memory_limit_check(size_t size, int gentle) { static size_t limit = 0; -- gitgitgadget ^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v6 2/6] core.fsyncmethod: add writeout-only mode 2022-03-10 22:43 ` [PATCH v6 0/6] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget 2022-03-10 22:43 ` [PATCH v6 1/6] wrapper: make inclusion of Windows csprng header tightly scoped Neeraj Singh via GitGitGadget @ 2022-03-10 22:43 ` Neeraj Singh via GitGitGadget 2022-03-10 22:43 ` [PATCH v6 3/6] core.fsync: introduce granular fsync control infrastructure Neeraj Singh via GitGitGadget ` (5 subsequent siblings) 7 siblings, 0 replies; 122+ messages in thread From: Neeraj Singh via GitGitGadget @ 2022-03-10 22:43 UTC (permalink / raw) To: git Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps, sandals, Johannes Schindelin, Neeraj K. Singh, Neeraj Singh From: Neeraj Singh <neerajsi@microsoft.com> This commit introduces the `core.fsyncMethod` configuration knob, which can currently be set to `fsync` or `writeout-only`. The new writeout-only mode attempts to tell the operating system to flush its in-memory page cache to the storage hardware without issuing a CACHE_FLUSH command to the storage controller. Writeout-only fsync is significantly faster than a vanilla fsync on common hardware, since data is written to a disk-side cache rather than all the way to a durable medium. Later changes in this patch series will take advantage of this primitive to implement batching of hardware flushes. When git_fsync is called with FSYNC_WRITEOUT_ONLY, it may fail and the caller is expected to do an ordinary fsync as needed. On Apple platforms, the fsync system call does not issue a CACHE_FLUSH directive to the storage controller. This change updates fsync to do fcntl(F_FULLFSYNC) to make fsync actually durable. We maintain parity with existing behavior on Apple platforms by setting the default value of the new core.fsyncMethod option. Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> --- Documentation/config/core.txt | 9 ++++ Makefile | 6 +++ cache.h | 7 ++++ compat/mingw.h | 3 ++ compat/win32/flush.c | 28 +++++++++++++ config.c | 12 ++++++ config.mak.uname | 3 ++ configure.ac | 8 ++++ contrib/buildsystems/CMakeLists.txt | 16 ++++++-- environment.c | 1 + git-compat-util.h | 24 +++++++++++ wrapper.c | 64 +++++++++++++++++++++++++++++ write-or-die.c | 11 +++-- 13 files changed, 184 insertions(+), 8 deletions(-) create mode 100644 compat/win32/flush.c diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt index c04f62a54a1..dbb134f7136 100644 --- a/Documentation/config/core.txt +++ b/Documentation/config/core.txt @@ -547,6 +547,15 @@ core.whitespace:: is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent` errors. The default tab width is 8. Allowed values are 1 to 63. +core.fsyncMethod:: + A value indicating the strategy Git will use to harden repository data + using fsync and related primitives. ++ +* `fsync` uses the fsync() system call or platform equivalents. +* `writeout-only` issues pagecache writeback requests, but depending on the + filesystem and storage hardware, data added to the repository may not be + durable in the event of a system crash. This is the default mode on macOS. + core.fsyncObjectFiles:: This boolean will enable 'fsync()' when writing object files. + diff --git a/Makefile b/Makefile index 6f0b4b775fe..17fd9b023a4 100644 --- a/Makefile +++ b/Makefile @@ -411,6 +411,8 @@ all:: # # Define HAVE_CLOCK_MONOTONIC if your platform has CLOCK_MONOTONIC. # +# Define HAVE_SYNC_FILE_RANGE if your platform has sync_file_range. +# # Define NEEDS_LIBRT if your platform requires linking with librt (glibc version # before 2.17) for clock_gettime and CLOCK_MONOTONIC. # @@ -1897,6 +1899,10 @@ ifdef HAVE_CLOCK_MONOTONIC BASIC_CFLAGS += -DHAVE_CLOCK_MONOTONIC endif +ifdef HAVE_SYNC_FILE_RANGE + BASIC_CFLAGS += -DHAVE_SYNC_FILE_RANGE +endif + ifdef NEEDS_LIBRT EXTLIBS += -lrt endif diff --git a/cache.h b/cache.h index 04d4d2db25c..82f0194a3dd 100644 --- a/cache.h +++ b/cache.h @@ -995,6 +995,13 @@ extern char *git_replace_ref_base; extern int fsync_object_files; extern int use_fsync; + +enum fsync_method { + FSYNC_METHOD_FSYNC, + FSYNC_METHOD_WRITEOUT_ONLY +}; + +extern enum fsync_method fsync_method; extern int core_preload_index; extern int precomposed_unicode; extern int protect_hfs; diff --git a/compat/mingw.h b/compat/mingw.h index c9a52ad64a6..6074a3d3ced 100644 --- a/compat/mingw.h +++ b/compat/mingw.h @@ -329,6 +329,9 @@ int mingw_getpagesize(void); #define getpagesize mingw_getpagesize #endif +int win32_fsync_no_flush(int fd); +#define fsync_no_flush win32_fsync_no_flush + struct rlimit { unsigned int rlim_cur; }; diff --git a/compat/win32/flush.c b/compat/win32/flush.c new file mode 100644 index 00000000000..291f90ea940 --- /dev/null +++ b/compat/win32/flush.c @@ -0,0 +1,28 @@ +#include "git-compat-util.h" +#include <winternl.h> +#include "lazyload.h" + +int win32_fsync_no_flush(int fd) +{ + IO_STATUS_BLOCK io_status; + +#define FLUSH_FLAGS_FILE_DATA_ONLY 1 + + DECLARE_PROC_ADDR(ntdll.dll, NTSTATUS, NTAPI, NtFlushBuffersFileEx, + HANDLE FileHandle, ULONG Flags, PVOID Parameters, ULONG ParameterSize, + PIO_STATUS_BLOCK IoStatusBlock); + + if (!INIT_PROC_ADDR(NtFlushBuffersFileEx)) { + errno = ENOSYS; + return -1; + } + + memset(&io_status, 0, sizeof(io_status)); + if (NtFlushBuffersFileEx((HANDLE)_get_osfhandle(fd), FLUSH_FLAGS_FILE_DATA_ONLY, + NULL, 0, &io_status)) { + errno = EINVAL; + return -1; + } + + return 0; +} diff --git a/config.c b/config.c index 383b1a4885b..f3ff80b01c9 100644 --- a/config.c +++ b/config.c @@ -1600,6 +1600,18 @@ static int git_default_core_config(const char *var, const char *value, void *cb) return 0; } + if (!strcmp(var, "core.fsyncmethod")) { + if (!value) + return config_error_nonbool(var); + if (!strcmp(value, "fsync")) + fsync_method = FSYNC_METHOD_FSYNC; + else if (!strcmp(value, "writeout-only")) + fsync_method = FSYNC_METHOD_WRITEOUT_ONLY; + else + warning(_("ignoring unknown core.fsyncMethod value '%s'"), value); + + } + if (!strcmp(var, "core.fsyncobjectfiles")) { fsync_object_files = git_config_bool(var, value); return 0; diff --git a/config.mak.uname b/config.mak.uname index 4352ea39e9b..404fff5dd04 100644 --- a/config.mak.uname +++ b/config.mak.uname @@ -57,6 +57,7 @@ ifeq ($(uname_S),Linux) HAVE_CLOCK_MONOTONIC = YesPlease # -lrt is needed for clock_gettime on glibc <= 2.16 NEEDS_LIBRT = YesPlease + HAVE_SYNC_FILE_RANGE = YesPlease HAVE_GETDELIM = YesPlease FREAD_READS_DIRECTORIES = UnfortunatelyYes BASIC_CFLAGS += -DHAVE_SYSINFO @@ -463,6 +464,7 @@ endif CFLAGS = BASIC_CFLAGS = -nologo -I. -Icompat/vcbuild/include -DWIN32 -D_CONSOLE -DHAVE_STRING_H -D_CRT_SECURE_NO_WARNINGS -D_CRT_NONSTDC_NO_DEPRECATE COMPAT_OBJS = compat/msvc.o compat/winansi.o \ + compat/win32/flush.o \ compat/win32/path-utils.o \ compat/win32/pthread.o compat/win32/syslog.o \ compat/win32/trace2_win32_process_info.o \ @@ -640,6 +642,7 @@ ifeq ($(uname_S),MINGW) COMPAT_CFLAGS += -DSTRIP_EXTENSION=\".exe\" COMPAT_OBJS += compat/mingw.o compat/winansi.o \ compat/win32/trace2_win32_process_info.o \ + compat/win32/flush.o \ compat/win32/path-utils.o \ compat/win32/pthread.o compat/win32/syslog.o \ compat/win32/dirent.o diff --git a/configure.ac b/configure.ac index 5ee25ec95c8..6bd6bef1c44 100644 --- a/configure.ac +++ b/configure.ac @@ -1082,6 +1082,14 @@ AC_COMPILE_IFELSE([CLOCK_MONOTONIC_SRC], [AC_MSG_RESULT([no]) HAVE_CLOCK_MONOTONIC=]) GIT_CONF_SUBST([HAVE_CLOCK_MONOTONIC]) + +# +# Define HAVE_SYNC_FILE_RANGE=YesPlease if sync_file_range is available. +GIT_CHECK_FUNC(sync_file_range, + [HAVE_SYNC_FILE_RANGE=YesPlease], + [HAVE_SYNC_FILE_RANGE]) +GIT_CONF_SUBST([HAVE_SYNC_FILE_RANGE]) + # # Define NO_SETITIMER if you don't have setitimer. GIT_CHECK_FUNC(setitimer, diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt index e44232f85d3..3a9e6241660 100644 --- a/contrib/buildsystems/CMakeLists.txt +++ b/contrib/buildsystems/CMakeLists.txt @@ -261,10 +261,18 @@ if(CMAKE_SYSTEM_NAME STREQUAL "Windows") NOGDI OBJECT_CREATION_MODE=1 __USE_MINGW_ANSI_STDIO=0 USE_NED_ALLOCATOR OVERRIDE_STRDUP MMAP_PREVENTS_DELETE USE_WIN32_MMAP UNICODE _UNICODE HAVE_WPGMPTR ENSURE_MSYSTEM_IS_SET HAVE_RTLGENRANDOM) - list(APPEND compat_SOURCES compat/mingw.c compat/winansi.c compat/win32/path-utils.c - compat/win32/pthread.c compat/win32mmap.c compat/win32/syslog.c - compat/win32/trace2_win32_process_info.c compat/win32/dirent.c - compat/nedmalloc/nedmalloc.c compat/strdup.c) + list(APPEND compat_SOURCES + compat/mingw.c + compat/winansi.c + compat/win32/flush.c + compat/win32/path-utils.c + compat/win32/pthread.c + compat/win32mmap.c + compat/win32/syslog.c + compat/win32/trace2_win32_process_info.c + compat/win32/dirent.c + compat/nedmalloc/nedmalloc.c + compat/strdup.c) set(NO_UNIX_SOCKETS 1) elseif(CMAKE_SYSTEM_NAME STREQUAL "Linux") diff --git a/environment.c b/environment.c index fd0501e77a5..3e3620d759f 100644 --- a/environment.c +++ b/environment.c @@ -44,6 +44,7 @@ int zlib_compression_level = Z_BEST_SPEED; int pack_compression_level = Z_DEFAULT_COMPRESSION; int fsync_object_files; int use_fsync = -1; +enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT; size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE; size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT; size_t delta_base_cache_limit = 96 * 1024 * 1024; diff --git a/git-compat-util.h b/git-compat-util.h index d210cff058c..00356476a9d 100644 --- a/git-compat-util.h +++ b/git-compat-util.h @@ -1271,6 +1271,30 @@ __attribute__((format (printf, 1, 2))) NORETURN void BUG(const char *fmt, ...); #endif +#ifdef __APPLE__ +#define FSYNC_METHOD_DEFAULT FSYNC_METHOD_WRITEOUT_ONLY +#else +#define FSYNC_METHOD_DEFAULT FSYNC_METHOD_FSYNC +#endif + +enum fsync_action { + FSYNC_WRITEOUT_ONLY, + FSYNC_HARDWARE_FLUSH +}; + +/* + * Issues an fsync against the specified file according to the specified mode. + * + * FSYNC_WRITEOUT_ONLY attempts to use interfaces available on some operating + * systems to flush the OS cache without issuing a flush command to the storage + * controller. If those interfaces are unavailable, the function fails with + * ENOSYS. + * + * FSYNC_HARDWARE_FLUSH does an OS writeout and hardware flush to ensure that + * changes are durable. It is not expected to fail. + */ +int git_fsync(int fd, enum fsync_action action); + /* * Preserves errno, prints a message, but gives no warning for ENOENT. * Returns 0 on success, which includes trying to unlink an object that does diff --git a/wrapper.c b/wrapper.c index 1108e4840a4..354d784c034 100644 --- a/wrapper.c +++ b/wrapper.c @@ -546,6 +546,70 @@ int xmkstemp_mode(char *filename_template, int mode) return fd; } +/* + * Some platforms return EINTR from fsync. Since fsync is invoked in some + * cases by a wrapper that dies on failure, do not expose EINTR to callers. + */ +static int fsync_loop(int fd) +{ + int err; + + do { + err = fsync(fd); + } while (err < 0 && errno == EINTR); + return err; +} + +int git_fsync(int fd, enum fsync_action action) +{ + switch (action) { + case FSYNC_WRITEOUT_ONLY: + +#ifdef __APPLE__ + /* + * On macOS, fsync just causes filesystem cache writeback but + * does not flush hardware caches. + */ + return fsync_loop(fd); +#endif + +#ifdef HAVE_SYNC_FILE_RANGE + /* + * On linux 2.6.17 and above, sync_file_range is the way to + * issue a writeback without a hardware flush. An offset of + * 0 and size of 0 indicates writeout of the entire file and the + * wait flags ensure that all dirty data is written to the disk + * (potentially in a disk-side cache) before we continue. + */ + + return sync_file_range(fd, 0, 0, SYNC_FILE_RANGE_WAIT_BEFORE | + SYNC_FILE_RANGE_WRITE | + SYNC_FILE_RANGE_WAIT_AFTER); +#endif + +#ifdef fsync_no_flush + return fsync_no_flush(fd); +#endif + + errno = ENOSYS; + return -1; + + case FSYNC_HARDWARE_FLUSH: + /* + * On macOS, a special fcntl is required to really flush the + * caches within the storage controller. As of this writing, + * this is a very expensive operation on Apple SSDs. + */ +#ifdef __APPLE__ + return fcntl(fd, F_FULLFSYNC); +#else + return fsync_loop(fd); +#endif + default: + BUG("unexpected git_fsync(%d) call", action); + } +} + static int warn_if_unremovable(const char *op, const char *file, int rc) { int err; diff --git a/write-or-die.c b/write-or-die.c index a3d5784cec9..9faa5f9f563 100644 --- a/write-or-die.c +++ b/write-or-die.c @@ -62,10 +62,13 @@ void fsync_or_die(int fd, const char *msg) use_fsync = git_env_bool("GIT_TEST_FSYNC", 1); if (!use_fsync) return; - while (fsync(fd) < 0) { - if (errno != EINTR) - die_errno("fsync error on '%s'", msg); - } + + if (fsync_method == FSYNC_METHOD_WRITEOUT_ONLY && + git_fsync(fd, FSYNC_WRITEOUT_ONLY) >= 0) + return; + + if (git_fsync(fd, FSYNC_HARDWARE_FLUSH) < 0) + die_errno("fsync error on '%s'", msg); } void write_or_die(int fd, const void *buf, size_t count) -- gitgitgadget ^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v6 3/6] core.fsync: introduce granular fsync control infrastructure 2022-03-10 22:43 ` [PATCH v6 0/6] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget 2022-03-10 22:43 ` [PATCH v6 1/6] wrapper: make inclusion of Windows csprng header tightly scoped Neeraj Singh via GitGitGadget 2022-03-10 22:43 ` [PATCH v6 2/6] core.fsyncmethod: add writeout-only mode Neeraj Singh via GitGitGadget @ 2022-03-10 22:43 ` Neeraj Singh via GitGitGadget 2022-03-10 22:43 ` [PATCH v6 4/6] core.fsync: add configuration parsing Neeraj Singh via GitGitGadget ` (4 subsequent siblings) 7 siblings, 0 replies; 122+ messages in thread From: Neeraj Singh via GitGitGadget @ 2022-03-10 22:43 UTC (permalink / raw) To: git Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps, sandals, Johannes Schindelin, Neeraj K. Singh, Neeraj Singh From: Neeraj Singh <neerajsi@microsoft.com> This commit introduces the infrastructure for the core.fsync configuration knob. The repository components we want to sync are identified by flags so that we can turn on or off syncing for specific components. If core.fsyncObjectFiles is set and the core.fsync configuration also includes FSYNC_COMPONENT_LOOSE_OBJECT, we will fsync any loose objects. This picks the strictest data integrity behavior if core.fsync and core.fsyncObjectFiles are set to conflicting values. This change introduces the currently unused fsync_component helper, which will be used by a later patch that adds fsyncing to the refs backend. Actual configuration and documentation of the fsync components list are in other patches in the series to separate review of the underlying mechanism from the policy of how it's configured. Helped-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> --- builtin/fast-import.c | 2 +- builtin/index-pack.c | 4 ++-- builtin/pack-objects.c | 24 +++++++++++++++++------- bulk-checkin.c | 5 +++-- cache.h | 23 +++++++++++++++++++++++ commit-graph.c | 3 ++- csum-file.c | 5 +++-- csum-file.h | 3 ++- environment.c | 1 + midx.c | 3 ++- object-file.c | 13 +++++++++---- pack-bitmap-write.c | 3 ++- pack-write.c | 13 +++++++------ read-cache.c | 2 +- write-or-die.c | 26 ++++++++++++++++++++++---- 15 files changed, 97 insertions(+), 33 deletions(-) diff --git a/builtin/fast-import.c b/builtin/fast-import.c index b7105fcad9b..f2c036a8955 100644 --- a/builtin/fast-import.c +++ b/builtin/fast-import.c @@ -865,7 +865,7 @@ static void end_packfile(void) struct tag *t; close_pack_windows(pack_data); - finalize_hashfile(pack_file, cur_pack_oid.hash, 0); + finalize_hashfile(pack_file, cur_pack_oid.hash, FSYNC_COMPONENT_PACK, 0); fixup_pack_header_footer(pack_data->pack_fd, pack_data->hash, pack_data->pack_name, object_count, cur_pack_oid.hash, pack_size); diff --git a/builtin/index-pack.c b/builtin/index-pack.c index c45273de3b1..c5f12f14df5 100644 --- a/builtin/index-pack.c +++ b/builtin/index-pack.c @@ -1290,7 +1290,7 @@ static void conclude_pack(int fix_thin_pack, const char *curr_pack, unsigned cha nr_objects - nr_objects_initial); stop_progress_msg(&progress, msg.buf); strbuf_release(&msg); - finalize_hashfile(f, tail_hash, 0); + finalize_hashfile(f, tail_hash, FSYNC_COMPONENT_PACK, 0); hashcpy(read_hash, pack_hash); fixup_pack_header_footer(output_fd, pack_hash, curr_pack, nr_objects, @@ -1512,7 +1512,7 @@ static void final(const char *final_pack_name, const char *curr_pack_name, if (!from_stdin) { close(input_fd); } else { - fsync_or_die(output_fd, curr_pack_name); + fsync_component_or_die(FSYNC_COMPONENT_PACK, output_fd, curr_pack_name); err = close(output_fd); if (err) die_errno(_("error while closing pack file")); diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 178e611f09d..c14fee8e99f 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -1199,16 +1199,26 @@ static void write_pack_file(void) display_progress(progress_state, written); } - /* - * Did we write the wrong # entries in the header? - * If so, rewrite it like in fast-import - */ if (pack_to_stdout) { - finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_CLOSE); + /* + * We never fsync when writing to stdout since we may + * not be writing to an actual pack file. For instance, + * the upload-pack code passes a pipe here. Calling + * fsync on a pipe results in unnecessary + * synchronization with the reader on some platforms. + */ + finalize_hashfile(f, hash, FSYNC_COMPONENT_NONE, + CSUM_HASH_IN_STREAM | CSUM_CLOSE); } else if (nr_written == nr_remaining) { - finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); + finalize_hashfile(f, hash, FSYNC_COMPONENT_PACK, + CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); } else { - int fd = finalize_hashfile(f, hash, 0); + /* + * If we wrote the wrong number of entries in the + * header, rewrite it like in fast-import. + */ + + int fd = finalize_hashfile(f, hash, FSYNC_COMPONENT_PACK, 0); fixup_pack_header_footer(fd, hash, pack_tmp_name, nr_written, hash, offset); close(fd); diff --git a/bulk-checkin.c b/bulk-checkin.c index 8785b2ac806..a2cf9dcbc8d 100644 --- a/bulk-checkin.c +++ b/bulk-checkin.c @@ -53,9 +53,10 @@ static void finish_bulk_checkin(struct bulk_checkin_state *state) unlink(state->pack_tmp_name); goto clear_exit; } else if (state->nr_written == 1) { - finalize_hashfile(state->f, hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); + finalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK, + CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); } else { - int fd = finalize_hashfile(state->f, hash, 0); + int fd = finalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK, 0); fixup_pack_header_footer(fd, hash, state->pack_tmp_name, state->nr_written, hash, state->offset); diff --git a/cache.h b/cache.h index 82f0194a3dd..7ac1959258d 100644 --- a/cache.h +++ b/cache.h @@ -993,6 +993,27 @@ void reset_shared_repository(void); extern int read_replace_refs; extern char *git_replace_ref_base; +/* + * These values are used to help identify parts of a repository to fsync. + * FSYNC_COMPONENT_NONE identifies data that will not be a persistent part of the + * repository and so shouldn't be fsynced. + */ +enum fsync_component { + FSYNC_COMPONENT_NONE, + FSYNC_COMPONENT_LOOSE_OBJECT = 1 << 0, + FSYNC_COMPONENT_PACK = 1 << 1, + FSYNC_COMPONENT_PACK_METADATA = 1 << 2, + FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3, +}; + +#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \ + FSYNC_COMPONENT_PACK_METADATA | \ + FSYNC_COMPONENT_COMMIT_GRAPH) + +/* + * A bitmask indicating which components of the repo should be fsynced. + */ +extern enum fsync_component fsync_components; extern int fsync_object_files; extern int use_fsync; @@ -1707,6 +1728,8 @@ int copy_file_with_time(const char *dst, const char *src, int mode); void write_or_die(int fd, const void *buf, size_t count); void fsync_or_die(int fd, const char *); +int fsync_component(enum fsync_component component, int fd); +void fsync_component_or_die(enum fsync_component component, int fd, const char *msg); ssize_t read_in_full(int fd, void *buf, size_t count); ssize_t write_in_full(int fd, const void *buf, size_t count); diff --git a/commit-graph.c b/commit-graph.c index 265c010122e..64897f57d9f 100644 --- a/commit-graph.c +++ b/commit-graph.c @@ -1942,7 +1942,8 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx) } close_commit_graph(ctx->r->objects); - finalize_hashfile(f, file_hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC); + finalize_hashfile(f, file_hash, FSYNC_COMPONENT_COMMIT_GRAPH, + CSUM_HASH_IN_STREAM | CSUM_FSYNC); free_chunkfile(cf); if (ctx->split) { diff --git a/csum-file.c b/csum-file.c index 26e8a6df44e..59ef3398ca2 100644 --- a/csum-file.c +++ b/csum-file.c @@ -58,7 +58,8 @@ static void free_hashfile(struct hashfile *f) free(f); } -int finalize_hashfile(struct hashfile *f, unsigned char *result, unsigned int flags) +int finalize_hashfile(struct hashfile *f, unsigned char *result, + enum fsync_component component, unsigned int flags) { int fd; @@ -69,7 +70,7 @@ int finalize_hashfile(struct hashfile *f, unsigned char *result, unsigned int fl if (flags & CSUM_HASH_IN_STREAM) flush(f, f->buffer, the_hash_algo->rawsz); if (flags & CSUM_FSYNC) - fsync_or_die(f->fd, f->name); + fsync_component_or_die(component, f->fd, f->name); if (flags & CSUM_CLOSE) { if (close(f->fd)) die_errno("%s: sha1 file error on close", f->name); diff --git a/csum-file.h b/csum-file.h index 291215b34eb..0d29f528fbc 100644 --- a/csum-file.h +++ b/csum-file.h @@ -1,6 +1,7 @@ #ifndef CSUM_FILE_H #define CSUM_FILE_H +#include "cache.h" #include "hash.h" struct progress; @@ -38,7 +39,7 @@ int hashfile_truncate(struct hashfile *, struct hashfile_checkpoint *); struct hashfile *hashfd(int fd, const char *name); struct hashfile *hashfd_check(const char *name); struct hashfile *hashfd_throughput(int fd, const char *name, struct progress *tp); -int finalize_hashfile(struct hashfile *, unsigned char *, unsigned int); +int finalize_hashfile(struct hashfile *, unsigned char *, enum fsync_component, unsigned int); void hashwrite(struct hashfile *, const void *, unsigned int); void hashflush(struct hashfile *f); void crc32_begin(struct hashfile *); diff --git a/environment.c b/environment.c index 3e3620d759f..36ca5fb2e77 100644 --- a/environment.c +++ b/environment.c @@ -45,6 +45,7 @@ int pack_compression_level = Z_DEFAULT_COMPRESSION; int fsync_object_files; int use_fsync = -1; enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT; +enum fsync_component fsync_components = FSYNC_COMPONENTS_DEFAULT; size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE; size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT; size_t delta_base_cache_limit = 96 * 1024 * 1024; diff --git a/midx.c b/midx.c index 865170bad05..107365d2114 100644 --- a/midx.c +++ b/midx.c @@ -1438,7 +1438,8 @@ static int write_midx_internal(const char *object_dir, write_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs); write_chunkfile(cf, &ctx); - finalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM); + finalize_hashfile(f, midx_hash, FSYNC_COMPONENT_PACK_METADATA, + CSUM_FSYNC | CSUM_HASH_IN_STREAM); free_chunkfile(cf); if (flags & MIDX_WRITE_REV_INDEX && diff --git a/object-file.c b/object-file.c index 03bd6a3baf3..e3f0bf27ff1 100644 --- a/object-file.c +++ b/object-file.c @@ -1849,11 +1849,16 @@ int hash_object_file(const struct git_hash_algo *algo, const void *buf, /* Finalize a file on disk, and close it. */ static void close_loose_object(int fd) { - if (!the_repository->objects->odb->will_destroy) { - if (fsync_object_files) - fsync_or_die(fd, "loose object file"); - } + if (the_repository->objects->odb->will_destroy) + goto out; + if (fsync_object_files > 0) + fsync_or_die(fd, "loose object file"); + else + fsync_component_or_die(FSYNC_COMPONENT_LOOSE_OBJECT, fd, + "loose object file"); + +out: if (close(fd) != 0) die_errno(_("error when closing loose object file")); } diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c index cab3eaa2acd..cf681547f2e 100644 --- a/pack-bitmap-write.c +++ b/pack-bitmap-write.c @@ -719,7 +719,8 @@ void bitmap_writer_finish(struct pack_idx_entry **index, if (options & BITMAP_OPT_HASH_CACHE) write_hash_cache(f, index, index_nr); - finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); + finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA, + CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); if (adjust_shared_perm(tmp_file.buf)) die_errno("unable to make temporary bitmap file readable"); diff --git a/pack-write.c b/pack-write.c index a5846f3a346..51812cb1299 100644 --- a/pack-write.c +++ b/pack-write.c @@ -159,9 +159,9 @@ const char *write_idx_file(const char *index_name, struct pack_idx_entry **objec } hashwrite(f, sha1, the_hash_algo->rawsz); - finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE | - ((opts->flags & WRITE_IDX_VERIFY) - ? 0 : CSUM_FSYNC)); + finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA, + CSUM_HASH_IN_STREAM | CSUM_CLOSE | + ((opts->flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC)); return index_name; } @@ -281,8 +281,9 @@ const char *write_rev_file_order(const char *rev_name, if (rev_name && adjust_shared_perm(rev_name) < 0) die(_("failed to make %s readable"), rev_name); - finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE | - ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC)); + finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA, + CSUM_HASH_IN_STREAM | CSUM_CLOSE | + ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC)); return rev_name; } @@ -390,7 +391,7 @@ void fixup_pack_header_footer(int pack_fd, the_hash_algo->final_fn(partial_pack_hash, &old_hash_ctx); the_hash_algo->final_fn(new_pack_hash, &new_hash_ctx); write_or_die(pack_fd, new_pack_hash, the_hash_algo->rawsz); - fsync_or_die(pack_fd, pack_name); + fsync_component_or_die(FSYNC_COMPONENT_PACK, pack_fd, pack_name); } char *index_pack_lockfile(int ip_out, int *is_well_formed) diff --git a/read-cache.c b/read-cache.c index 79b9b99ebf7..df869691fd4 100644 --- a/read-cache.c +++ b/read-cache.c @@ -3089,7 +3089,7 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile, return -1; } - finalize_hashfile(f, istate->oid.hash, CSUM_HASH_IN_STREAM); + finalize_hashfile(f, istate->oid.hash, FSYNC_COMPONENT_NONE, CSUM_HASH_IN_STREAM); if (close_tempfile_gently(tempfile)) { error(_("could not close '%s'"), get_tempfile_path(tempfile)); return -1; diff --git a/write-or-die.c b/write-or-die.c index 9faa5f9f563..c4fd91b5b43 100644 --- a/write-or-die.c +++ b/write-or-die.c @@ -56,21 +56,39 @@ void fprintf_or_die(FILE *f, const char *fmt, ...) } } -void fsync_or_die(int fd, const char *msg) +static int maybe_fsync(int fd) { if (use_fsync < 0) use_fsync = git_env_bool("GIT_TEST_FSYNC", 1); if (!use_fsync) - return; + return 0; if (fsync_method == FSYNC_METHOD_WRITEOUT_ONLY && git_fsync(fd, FSYNC_WRITEOUT_ONLY) >= 0) - return; + return 0; + + return git_fsync(fd, FSYNC_HARDWARE_FLUSH); +} - if (git_fsync(fd, FSYNC_HARDWARE_FLUSH) < 0) +void fsync_or_die(int fd, const char *msg) +{ + if (maybe_fsync(fd) < 0) die_errno("fsync error on '%s'", msg); } +int fsync_component(enum fsync_component component, int fd) +{ + if (fsync_components & component) + return maybe_fsync(fd); + return 0; +} + +void fsync_component_or_die(enum fsync_component component, int fd, const char *msg) +{ + if (fsync_components & component) + fsync_or_die(fd, msg); +} + void write_or_die(int fd, const void *buf, size_t count) { if (write_in_full(fd, buf, count) < 0) { -- gitgitgadget ^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v6 4/6] core.fsync: add configuration parsing 2022-03-10 22:43 ` [PATCH v6 0/6] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget ` (2 preceding siblings ...) 2022-03-10 22:43 ` [PATCH v6 3/6] core.fsync: introduce granular fsync control infrastructure Neeraj Singh via GitGitGadget @ 2022-03-10 22:43 ` Neeraj Singh via GitGitGadget 2022-03-28 11:06 ` Jiang Xin 2022-03-10 22:43 ` [PATCH v6 5/6] core.fsync: new option to harden the index Neeraj Singh via GitGitGadget ` (3 subsequent siblings) 7 siblings, 1 reply; 122+ messages in thread From: Neeraj Singh via GitGitGadget @ 2022-03-10 22:43 UTC (permalink / raw) To: git Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps, sandals, Johannes Schindelin, Neeraj K. Singh, Neeraj Singh From: Neeraj Singh <neerajsi@microsoft.com> This change introduces code to parse the core.fsync setting and configure the fsync_components variable. core.fsync is configured as a comma-separated list of component names to sync. Each time a core.fsync variable is encountered in the configuration heirarchy, we start off with a clean state with the platform default value. Passing 'none' resets the value to indicate nothing will be synced. We gather all negative and positive entries from the comma separated list and then compute the new value by removing all the negative entries and adding all of the positive entries. We issue a warning for components that are not recognized so that the configuration code is compatible with configs from future versions of Git with more repo components. Complete documentation for the new setting is included in a later patch in the series so that it can be reviewed once in final form. Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> --- Documentation/config/core.txt | 9 +++-- config.c | 76 +++++++++++++++++++++++++++++++++++ environment.c | 2 +- 3 files changed, 82 insertions(+), 5 deletions(-) diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt index dbb134f7136..ab911d6e269 100644 --- a/Documentation/config/core.txt +++ b/Documentation/config/core.txt @@ -558,11 +558,12 @@ core.fsyncMethod:: core.fsyncObjectFiles:: This boolean will enable 'fsync()' when writing object files. + This setting is deprecated. Use core.fsync instead. + -This is a total waste of time and effort on a filesystem that orders -data writes properly, but can be useful for filesystems that do not use -journalling (traditional UNIX filesystems) or that only journal metadata -and not file contents (OS X's HFS+, or Linux ext3 with "data=writeback"). +This setting affects data added to the Git repository in loose-object +form. When set to true, Git will issue an fsync or similar system call +to flush caches so that loose-objects remain consistent in the face +of a unclean system shutdown. core.preloadIndex:: Enable parallel index preload for operations like 'git diff' diff --git a/config.c b/config.c index f3ff80b01c9..94a4598b5fc 100644 --- a/config.c +++ b/config.c @@ -1323,6 +1323,73 @@ static int git_parse_maybe_bool_text(const char *value) return -1; } +static const struct fsync_component_name { + const char *name; + enum fsync_component component_bits; +} fsync_component_names[] = { + { "loose-object", FSYNC_COMPONENT_LOOSE_OBJECT }, + { "pack", FSYNC_COMPONENT_PACK }, + { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA }, + { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, +}; + +static enum fsync_component parse_fsync_components(const char *var, const char *string) +{ + enum fsync_component current = FSYNC_COMPONENTS_DEFAULT; + enum fsync_component positive = 0, negative = 0; + + while (string) { + int i; + size_t len; + const char *ep; + int negated = 0; + int found = 0; + + string = string + strspn(string, ", \t\n\r"); + ep = strchrnul(string, ','); + len = ep - string; + if (!strcmp(string, "none")) { + current = FSYNC_COMPONENT_NONE; + goto next_name; + } + + if (*string == '-') { + negated = 1; + string++; + len--; + if (!len) + warning(_("invalid value for variable %s"), var); + } + + if (!len) + break; + + for (i = 0; i < ARRAY_SIZE(fsync_component_names); ++i) { + const struct fsync_component_name *n = &fsync_component_names[i]; + + if (strncmp(n->name, string, len)) + continue; + + found = 1; + if (negated) + negative |= n->component_bits; + else + positive |= n->component_bits; + } + + if (!found) { + char *component = xstrndup(string, len); + warning(_("ignoring unknown core.fsync component '%s'"), component); + free(component); + } + +next_name: + string = ep; + } + + return (current & ~negative) | positive; +} + int git_parse_maybe_bool(const char *value) { int v = git_parse_maybe_bool_text(value); @@ -1600,6 +1667,13 @@ static int git_default_core_config(const char *var, const char *value, void *cb) return 0; } + if (!strcmp(var, "core.fsync")) { + if (!value) + return config_error_nonbool(var); + fsync_components = parse_fsync_components(var, value); + return 0; + } + if (!strcmp(var, "core.fsyncmethod")) { if (!value) return config_error_nonbool(var); @@ -1613,6 +1687,8 @@ static int git_default_core_config(const char *var, const char *value, void *cb) } if (!strcmp(var, "core.fsyncobjectfiles")) { + if (fsync_object_files < 0) + warning(_("core.fsyncobjectfiles is deprecated; use core.fsync instead")); fsync_object_files = git_config_bool(var, value); return 0; } diff --git a/environment.c b/environment.c index 36ca5fb2e77..698f03a2f47 100644 --- a/environment.c +++ b/environment.c @@ -42,7 +42,7 @@ const char *git_attributes_file; const char *git_hooks_path; int zlib_compression_level = Z_BEST_SPEED; int pack_compression_level = Z_DEFAULT_COMPRESSION; -int fsync_object_files; +int fsync_object_files = -1; int use_fsync = -1; enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT; enum fsync_component fsync_components = FSYNC_COMPONENTS_DEFAULT; -- gitgitgadget ^ permalink raw reply related [flat|nested] 122+ messages in thread
* Re: [PATCH v6 4/6] core.fsync: add configuration parsing 2022-03-10 22:43 ` [PATCH v6 4/6] core.fsync: add configuration parsing Neeraj Singh via GitGitGadget @ 2022-03-28 11:06 ` Jiang Xin 2022-03-28 19:45 ` Neeraj Singh 0 siblings, 1 reply; 122+ messages in thread From: Jiang Xin @ 2022-03-28 11:06 UTC (permalink / raw) To: Neeraj Singh via GitGitGadget Cc: Git List, rsbecker, Bagas Sanjaya, Elijah Newren, Ævar Arnfjörð Bjarmason, nksingh85, ps, brian m. carlson, Johannes Schindelin, Neeraj K. Singh On Sat, Mar 12, 2022 at 6:25 AM Neeraj Singh via GitGitGadget <gitgitgadget@gmail.com> wrote: > @@ -1613,6 +1687,8 @@ static int git_default_core_config(const char *var, const char *value, void *cb) > } > > if (!strcmp(var, "core.fsyncobjectfiles")) { > + if (fsync_object_files < 0) > + warning(_("core.fsyncobjectfiles is deprecated; use core.fsync instead")); s/core.fsyncobjectfiles/core.fsyncObjectFiles/ to use bumpyCaps for config variable in documentation. ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v6 4/6] core.fsync: add configuration parsing 2022-03-28 11:06 ` Jiang Xin @ 2022-03-28 19:45 ` Neeraj Singh 0 siblings, 0 replies; 122+ messages in thread From: Neeraj Singh @ 2022-03-28 19:45 UTC (permalink / raw) To: Jiang Xin Cc: Neeraj Singh via GitGitGadget, Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Ævar Arnfjörð Bjarmason, Patrick Steinhardt, brian m. carlson, Johannes Schindelin, Neeraj K. Singh On Mon, Mar 28, 2022 at 4:07 AM Jiang Xin <worldhello.net@gmail.com> wrote: > > On Sat, Mar 12, 2022 at 6:25 AM Neeraj Singh via GitGitGadget > <gitgitgadget@gmail.com> wrote: > > @@ -1613,6 +1687,8 @@ static int git_default_core_config(const char *var, const char *value, void *cb) > > } > > > > if (!strcmp(var, "core.fsyncobjectfiles")) { > > + if (fsync_object_files < 0) > > + warning(_("core.fsyncobjectfiles is deprecated; use core.fsync instead")); > > s/core.fsyncobjectfiles/core.fsyncObjectFiles/ to use bumpyCaps for > config variable in documentation. THanks for pointing this out. I'll fix it. ^ permalink raw reply [flat|nested] 122+ messages in thread
* [PATCH v6 5/6] core.fsync: new option to harden the index 2022-03-10 22:43 ` [PATCH v6 0/6] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget ` (3 preceding siblings ...) 2022-03-10 22:43 ` [PATCH v6 4/6] core.fsync: add configuration parsing Neeraj Singh via GitGitGadget @ 2022-03-10 22:43 ` Neeraj Singh via GitGitGadget 2022-03-10 22:43 ` [PATCH v6 6/6] core.fsync: documentation and user-friendly aggregate options Neeraj Singh via GitGitGadget ` (2 subsequent siblings) 7 siblings, 0 replies; 122+ messages in thread From: Neeraj Singh via GitGitGadget @ 2022-03-10 22:43 UTC (permalink / raw) To: git Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps, sandals, Johannes Schindelin, Neeraj K. Singh, Neeraj Singh From: Neeraj Singh <neerajsi@microsoft.com> This commit introduces the new ability for the user to harden the index. In the event of a system crash, the index must be durable for the user to actually find a file that has been added to the repo and then deleted from the working tree. We use the presence of the COMMIT_LOCK flag and absence of the alternate_index_output as a proxy for determining whether we're updating the persistent index of the repo or some temporary index. We don't sync these temporary indexes. Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> --- cache.h | 1 + config.c | 1 + read-cache.c | 19 +++++++++++++------ 3 files changed, 15 insertions(+), 6 deletions(-) diff --git a/cache.h b/cache.h index 7ac1959258d..e08eeac6c15 100644 --- a/cache.h +++ b/cache.h @@ -1004,6 +1004,7 @@ enum fsync_component { FSYNC_COMPONENT_PACK = 1 << 1, FSYNC_COMPONENT_PACK_METADATA = 1 << 2, FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3, + FSYNC_COMPONENT_INDEX = 1 << 4, }; #define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \ diff --git a/config.c b/config.c index 94a4598b5fc..80f33c91982 100644 --- a/config.c +++ b/config.c @@ -1331,6 +1331,7 @@ static const struct fsync_component_name { { "pack", FSYNC_COMPONENT_PACK }, { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA }, { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, + { "index", FSYNC_COMPONENT_INDEX }, }; static enum fsync_component parse_fsync_components(const char *var, const char *string) diff --git a/read-cache.c b/read-cache.c index df869691fd4..7683b679258 100644 --- a/read-cache.c +++ b/read-cache.c @@ -2842,7 +2842,7 @@ static int record_ieot(void) * rely on it. */ static int do_write_index(struct index_state *istate, struct tempfile *tempfile, - int strip_extensions) + int strip_extensions, unsigned flags) { uint64_t start = getnanotime(); struct hashfile *f; @@ -2856,6 +2856,7 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile, struct strbuf previous_name_buf = STRBUF_INIT, *previous_name; int drop_cache_tree = istate->drop_cache_tree; off_t offset; + int csum_fsync_flag; int ieot_entries = 1; struct index_entry_offset_table *ieot = NULL; int nr, nr_threads; @@ -3089,7 +3090,13 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile, return -1; } - finalize_hashfile(f, istate->oid.hash, FSYNC_COMPONENT_NONE, CSUM_HASH_IN_STREAM); + csum_fsync_flag = 0; + if (!alternate_index_output && (flags & COMMIT_LOCK)) + csum_fsync_flag = CSUM_FSYNC; + + finalize_hashfile(f, istate->oid.hash, FSYNC_COMPONENT_INDEX, + CSUM_HASH_IN_STREAM | csum_fsync_flag); + if (close_tempfile_gently(tempfile)) { error(_("could not close '%s'"), get_tempfile_path(tempfile)); return -1; @@ -3144,7 +3151,7 @@ static int do_write_locked_index(struct index_state *istate, struct lock_file *l */ trace2_region_enter_printf("index", "do_write_index", the_repository, "%s", get_lock_file_path(lock)); - ret = do_write_index(istate, lock->tempfile, 0); + ret = do_write_index(istate, lock->tempfile, 0, flags); trace2_region_leave_printf("index", "do_write_index", the_repository, "%s", get_lock_file_path(lock)); @@ -3238,7 +3245,7 @@ static int clean_shared_index_files(const char *current_hex) } static int write_shared_index(struct index_state *istate, - struct tempfile **temp) + struct tempfile **temp, unsigned flags) { struct split_index *si = istate->split_index; int ret, was_full = !istate->sparse_index; @@ -3248,7 +3255,7 @@ static int write_shared_index(struct index_state *istate, trace2_region_enter_printf("index", "shared/do_write_index", the_repository, "%s", get_tempfile_path(*temp)); - ret = do_write_index(si->base, *temp, 1); + ret = do_write_index(si->base, *temp, 1, flags); trace2_region_leave_printf("index", "shared/do_write_index", the_repository, "%s", get_tempfile_path(*temp)); @@ -3357,7 +3364,7 @@ int write_locked_index(struct index_state *istate, struct lock_file *lock, ret = do_write_locked_index(istate, lock, flags); goto out; } - ret = write_shared_index(istate, &temp); + ret = write_shared_index(istate, &temp, flags); saved_errno = errno; if (is_tempfile_active(temp)) -- gitgitgadget ^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v6 6/6] core.fsync: documentation and user-friendly aggregate options 2022-03-10 22:43 ` [PATCH v6 0/6] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget ` (4 preceding siblings ...) 2022-03-10 22:43 ` [PATCH v6 5/6] core.fsync: new option to harden the index Neeraj Singh via GitGitGadget @ 2022-03-10 22:43 ` Neeraj Singh via GitGitGadget 2022-03-15 19:12 ` [PATCH v7] " Neeraj Singh 2022-03-10 23:34 ` [PATCH v6 0/6] A design for future-proofing fsync() configuration Junio C Hamano 2022-03-11 9:58 ` [PATCH v2] core.fsync: new option to harden references Patrick Steinhardt 7 siblings, 1 reply; 122+ messages in thread From: Neeraj Singh via GitGitGadget @ 2022-03-10 22:43 UTC (permalink / raw) To: git Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, ps, sandals, Johannes Schindelin, Neeraj K. Singh, Neeraj Singh From: Neeraj Singh <neerajsi@microsoft.com> This commit adds aggregate options for the core.fsync setting that are more user-friendly. These options are specified in terms of 'levels of safety', indicating which Git operations are considered to be sync points for durability. The new documentation is also included here in its entirety for ease of review. Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> --- Documentation/config/core.txt | 40 +++++++++++++++++++++++++++++++++++ cache.h | 23 +++++++++++++++++--- config.c | 5 +++++ 3 files changed, 65 insertions(+), 3 deletions(-) diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt index ab911d6e269..37105a7be40 100644 --- a/Documentation/config/core.txt +++ b/Documentation/config/core.txt @@ -547,6 +547,46 @@ core.whitespace:: is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent` errors. The default tab width is 8. Allowed values are 1 to 63. +core.fsync:: + A comma-separated list of components of the repository that + should be hardened via the core.fsyncMethod when created or + modified. You can disable hardening of any component by + prefixing it with a '-'. Items that are not hardened may be + lost in the event of an unclean system shutdown. Unless you + have special requirements, it is recommended that you leave + this option empty or pick one of `committed`, `added`, + or `all`. ++ +When this configuration is encountered, the set of components starts with +the platform default value, disabled components are removed, and additional +components are added. `none` resets the state so that the platform default +is ignored. ++ +The empty string resets the fsync configuration to the platform +default. The platform default on most platform is equivalent to +`core.fsync=committed,-loose-object`, which has good performance, +but risks losing recent work in the event of an unclean system shutdown. ++ +* `none` clears the set of fsynced components. +* `loose-object` hardens objects added to the repo in loose-object form. +* `pack` hardens objects added to the repo in packfile form. +* `pack-metadata` hardens packfile bitmaps and indexes. +* `commit-graph` hardens the commit graph file. +* `index` hardens the index when it is modified. +* `objects` is an aggregate option that is equivalent to + `loose-object,pack`. +* `derived-metadata` is an aggregate option that is equivalent to + `pack-metadata,commit-graph`. +* `committed` is an aggregate option that is currently equivalent to + `objects`. This mode sacrifices some performance to ensure that work + that is committed to the repository with `git commit` or similar commands + is hardened. +* `added` is an aggregate option that is currently equivalent to + `committed,index`. This mode sacrifices additional performance to + ensure that the results of commands like `git add` and similar operations + are hardened. +* `all` is an aggregate option that syncs all individual components above. + core.fsyncMethod:: A value indicating the strategy Git will use to harden repository data using fsync and related primitives. diff --git a/cache.h b/cache.h index e08eeac6c15..86680f144ec 100644 --- a/cache.h +++ b/cache.h @@ -1007,9 +1007,26 @@ enum fsync_component { FSYNC_COMPONENT_INDEX = 1 << 4, }; -#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \ - FSYNC_COMPONENT_PACK_METADATA | \ - FSYNC_COMPONENT_COMMIT_GRAPH) +#define FSYNC_COMPONENTS_OBJECTS (FSYNC_COMPONENT_LOOSE_OBJECT | \ + FSYNC_COMPONENT_PACK) + +#define FSYNC_COMPONENTS_DERIVED_METADATA (FSYNC_COMPONENT_PACK_METADATA | \ + FSYNC_COMPONENT_COMMIT_GRAPH) + +#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENTS_OBJECTS | \ + FSYNC_COMPONENTS_DERIVED_METADATA | \ + ~FSYNC_COMPONENT_LOOSE_OBJECT) + +#define FSYNC_COMPONENTS_COMMITTED (FSYNC_COMPONENTS_OBJECTS) + +#define FSYNC_COMPONENTS_ADDED (FSYNC_COMPONENTS_COMMITTED | \ + FSYNC_COMPONENT_INDEX) + +#define FSYNC_COMPONENTS_ALL (FSYNC_COMPONENT_LOOSE_OBJECT | \ + FSYNC_COMPONENT_PACK | \ + FSYNC_COMPONENT_PACK_METADATA | \ + FSYNC_COMPONENT_COMMIT_GRAPH | \ + FSYNC_COMPONENT_INDEX) /* * A bitmask indicating which components of the repo should be fsynced. diff --git a/config.c b/config.c index 80f33c91982..fd8e1659312 100644 --- a/config.c +++ b/config.c @@ -1332,6 +1332,11 @@ static const struct fsync_component_name { { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA }, { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, { "index", FSYNC_COMPONENT_INDEX }, + { "objects", FSYNC_COMPONENTS_OBJECTS }, + { "derived-metadata", FSYNC_COMPONENTS_DERIVED_METADATA }, + { "committed", FSYNC_COMPONENTS_COMMITTED }, + { "added", FSYNC_COMPONENTS_ADDED }, + { "all", FSYNC_COMPONENTS_ALL }, }; static enum fsync_component parse_fsync_components(const char *var, const char *string) -- gitgitgadget ^ permalink raw reply related [flat|nested] 122+ messages in thread
* [PATCH v7] core.fsync: documentation and user-friendly aggregate options 2022-03-10 22:43 ` [PATCH v6 6/6] core.fsync: documentation and user-friendly aggregate options Neeraj Singh via GitGitGadget @ 2022-03-15 19:12 ` Neeraj Singh 2022-03-15 19:32 ` Junio C Hamano 2022-03-23 14:20 ` do we have too much fsync() configuration in 'next'? (was: [PATCH v7] core.fsync: documentation and user-friendly aggregate options) Ævar Arnfjörð Bjarmason 0 siblings, 2 replies; 122+ messages in thread From: Neeraj Singh @ 2022-03-15 19:12 UTC (permalink / raw) To: gitgitgadget Cc: Johannes.Schindelin, avarab, bagasdotme, git, neerajsi, newren, nksingh85, ps, rsbecker, sandals This commit adds aggregate options for the core.fsync setting that are more user-friendly. These options are specified in terms of 'levels of safety', indicating which Git operations are considered to be sync points for durability. The new documentation is also included here in its entirety for ease of review. Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> --- This revision fixes a grammatical mistake in the core.fsync documentation. Documentation/config/core.txt | 40 +++++++++++++++++++++++++++++++++++ cache.h | 23 +++++++++++++++++--- config.c | 5 +++++ 3 files changed, 65 insertions(+), 3 deletions(-) diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt index ab911d6e26..9a3ad71e9e 100644 --- a/Documentation/config/core.txt +++ b/Documentation/config/core.txt @@ -547,6 +547,46 @@ core.whitespace:: is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent` errors. The default tab width is 8. Allowed values are 1 to 63. +core.fsync:: + A comma-separated list of components of the repository that + should be hardened via the core.fsyncMethod when created or + modified. You can disable hardening of any component by + prefixing it with a '-'. Items that are not hardened may be + lost in the event of an unclean system shutdown. Unless you + have special requirements, it is recommended that you leave + this option empty or pick one of `committed`, `added`, + or `all`. ++ +When this configuration is encountered, the set of components starts with +the platform default value, disabled components are removed, and additional +components are added. `none` resets the state so that the platform default +is ignored. ++ +The empty string resets the fsync configuration to the platform +default. The default on most platforms is equivalent to +`core.fsync=committed,-loose-object`, which has good performance, +but risks losing recent work in the event of an unclean system shutdown. ++ +* `none` clears the set of fsynced components. +* `loose-object` hardens objects added to the repo in loose-object form. +* `pack` hardens objects added to the repo in packfile form. +* `pack-metadata` hardens packfile bitmaps and indexes. +* `commit-graph` hardens the commit graph file. +* `index` hardens the index when it is modified. +* `objects` is an aggregate option that is equivalent to + `loose-object,pack`. +* `derived-metadata` is an aggregate option that is equivalent to + `pack-metadata,commit-graph`. +* `committed` is an aggregate option that is currently equivalent to + `objects`. This mode sacrifices some performance to ensure that work + that is committed to the repository with `git commit` or similar commands + is hardened. +* `added` is an aggregate option that is currently equivalent to + `committed,index`. This mode sacrifices additional performance to + ensure that the results of commands like `git add` and similar operations + are hardened. +* `all` is an aggregate option that syncs all individual components above. + core.fsyncMethod:: A value indicating the strategy Git will use to harden repository data using fsync and related primitives. diff --git a/cache.h b/cache.h index e08eeac6c1..86680f144e 100644 --- a/cache.h +++ b/cache.h @@ -1007,9 +1007,26 @@ enum fsync_component { FSYNC_COMPONENT_INDEX = 1 << 4, }; -#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \ - FSYNC_COMPONENT_PACK_METADATA | \ - FSYNC_COMPONENT_COMMIT_GRAPH) +#define FSYNC_COMPONENTS_OBJECTS (FSYNC_COMPONENT_LOOSE_OBJECT | \ + FSYNC_COMPONENT_PACK) + +#define FSYNC_COMPONENTS_DERIVED_METADATA (FSYNC_COMPONENT_PACK_METADATA | \ + FSYNC_COMPONENT_COMMIT_GRAPH) + +#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENTS_OBJECTS | \ + FSYNC_COMPONENTS_DERIVED_METADATA | \ + ~FSYNC_COMPONENT_LOOSE_OBJECT) + +#define FSYNC_COMPONENTS_COMMITTED (FSYNC_COMPONENTS_OBJECTS) + +#define FSYNC_COMPONENTS_ADDED (FSYNC_COMPONENTS_COMMITTED | \ + FSYNC_COMPONENT_INDEX) + +#define FSYNC_COMPONENTS_ALL (FSYNC_COMPONENT_LOOSE_OBJECT | \ + FSYNC_COMPONENT_PACK | \ + FSYNC_COMPONENT_PACK_METADATA | \ + FSYNC_COMPONENT_COMMIT_GRAPH | \ + FSYNC_COMPONENT_INDEX) /* * A bitmask indicating which components of the repo should be fsynced. diff --git a/config.c b/config.c index 80f33c9198..fd8e165931 100644 --- a/config.c +++ b/config.c @@ -1332,6 +1332,11 @@ static const struct fsync_component_name { { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA }, { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, { "index", FSYNC_COMPONENT_INDEX }, + { "objects", FSYNC_COMPONENTS_OBJECTS }, + { "derived-metadata", FSYNC_COMPONENTS_DERIVED_METADATA }, + { "committed", FSYNC_COMPONENTS_COMMITTED }, + { "added", FSYNC_COMPONENTS_ADDED }, + { "all", FSYNC_COMPONENTS_ALL }, }; static enum fsync_component parse_fsync_components(const char *var, const char *string) -- 2.33.0.334.g55a40fc8fd ^ permalink raw reply related [flat|nested] 122+ messages in thread
* Re: [PATCH v7] core.fsync: documentation and user-friendly aggregate options 2022-03-15 19:12 ` [PATCH v7] " Neeraj Singh @ 2022-03-15 19:32 ` Junio C Hamano 2022-03-15 19:56 ` Neeraj Singh 2022-03-23 14:20 ` do we have too much fsync() configuration in 'next'? (was: [PATCH v7] core.fsync: documentation and user-friendly aggregate options) Ævar Arnfjörð Bjarmason 1 sibling, 1 reply; 122+ messages in thread From: Junio C Hamano @ 2022-03-15 19:32 UTC (permalink / raw) To: Neeraj Singh Cc: gitgitgadget, Johannes.Schindelin, avarab, bagasdotme, git, neerajsi, newren, ps, rsbecker, sandals Neeraj Singh <nksingh85@gmail.com> writes: > This commit adds aggregate options for the core.fsync setting that are > more user-friendly. These options are specified in terms of 'levels of > safety', indicating which Git operations are considered to be sync > points for durability. > > The new documentation is also included here in its entirety for ease of > review. > > Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> > --- > This revision fixes a grammatical mistake in the core.fsync documentation. Is this meant to be [PATCH v7 6/6] where 1-5/6 of v7 are supposed to be identical to their counterparts of v6 and therefor not sent? Not complaining, just double-checking, as I do not want to assume and end up missing the final updates to the other 5. In the meantime, I'd assume that it is the case, and will fix the author ident (you sent this from your gmail address) before replacing the last bit. The only change from the previous round is "the platform default on most platform" -> "the default on most platforms", which looks sensible. Thanks. 1: 39f4b94c2c ! 1: dfeab99d23 core.fsync: documentation and user-friendly aggregate options @@ ## Metadata ## -Author: Neeraj Singh <neerajsi@microsoft.com> +Author: Neeraj Singh <nksingh85@gmail.com> ## Commit message ## core.fsync: documentation and user-friendly aggregate options @@ Documentation/config/core.txt: core.whitespace:: +is ignored. ++ +The empty string resets the fsync configuration to the platform -+default. The platform default on most platform is equivalent to ++default. The default on most platforms is equivalent to +`core.fsync=committed,-loose-object`, which has good performance, +but risks losing recent work in the event of an unclean system shutdown. ++ Thanks. ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v7] core.fsync: documentation and user-friendly aggregate options 2022-03-15 19:32 ` Junio C Hamano @ 2022-03-15 19:56 ` Neeraj Singh 0 siblings, 0 replies; 122+ messages in thread From: Neeraj Singh @ 2022-03-15 19:56 UTC (permalink / raw) To: Junio C Hamano Cc: Neeraj K. Singh via GitGitGadget, Johannes Schindelin, Ævar Arnfjörð Bjarmason, Bagas Sanjaya, Git List, Neeraj Singh, Elijah Newren, Patrick Steinhardt, Randall S. Becker, brian m. carlson On Tue, Mar 15, 2022 at 12:32 PM Junio C Hamano <gitster@pobox.com> wrote: > > Neeraj Singh <nksingh85@gmail.com> writes: > > > This commit adds aggregate options for the core.fsync setting that are > > more user-friendly. These options are specified in terms of 'levels of > > safety', indicating which Git operations are considered to be sync > > points for durability. > > > > The new documentation is also included here in its entirety for ease of > > review. > > > > Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> > > --- > > This revision fixes a grammatical mistake in the core.fsync documentation. > > Is this meant to be [PATCH v7 6/6] where 1-5/6 of v7 are supposed to > be identical to their counterparts of v6 and therefor not sent? Not > complaining, just double-checking, as I do not want to assume and > end up missing the final updates to the other 5. > > In the meantime, I'd assume that it is the case, and will fix the > author ident (you sent this from your gmail address) before > replacing the last bit. > > The only change from the previous round is "the platform default on > most platform" -> "the default on most platforms", which looks > sensible. > > Thanks. Yes, that's right. I was trying out git-send-email for the first time. I didn't want to spam the list with a whole new series where only one patch had a change. > > 1: 39f4b94c2c ! 1: dfeab99d23 core.fsync: documentation and user-friendly aggregate options > @@ > ## Metadata ## > -Author: Neeraj Singh <neerajsi@microsoft.com> > +Author: Neeraj Singh <nksingh85@gmail.com> > > ## Commit message ## > core.fsync: documentation and user-friendly aggregate options > @@ Documentation/config/core.txt: core.whitespace:: > +is ignored. > ++ > +The empty string resets the fsync configuration to the platform > -+default. The platform default on most platform is equivalent to > ++default. The default on most platforms is equivalent to > +`core.fsync=committed,-loose-object`, which has good performance, > +but risks losing recent work in the event of an unclean system shutdown. > ++ > > Thanks. ^ permalink raw reply [flat|nested] 122+ messages in thread
* do we have too much fsync() configuration in 'next'? (was: [PATCH v7] core.fsync: documentation and user-friendly aggregate options) 2022-03-15 19:12 ` [PATCH v7] " Neeraj Singh 2022-03-15 19:32 ` Junio C Hamano @ 2022-03-23 14:20 ` Ævar Arnfjörð Bjarmason 2022-03-25 21:24 ` Neeraj Singh 1 sibling, 1 reply; 122+ messages in thread From: Ævar Arnfjörð Bjarmason @ 2022-03-23 14:20 UTC (permalink / raw) To: Neeraj Singh Cc: gitgitgadget, Johannes.Schindelin, bagasdotme, git, neerajsi, newren, ps, rsbecker, sandals On Tue, Mar 15 2022, Neeraj Singh wrote: I know this is probably 80% my fault by egging you on about initially adding the wildmatch() based thing you didn't go for. But having looked at this with fresh eyes quite deeply I really think we're severely over-configuring things here: > +core.fsync:: > + A comma-separated list of components of the repository that > + should be hardened via the core.fsyncMethod when created or > + modified. You can disable hardening of any component by > + prefixing it with a '-'. Items that are not hardened may be > + lost in the event of an unclean system shutdown. Unless you > + have special requirements, it is recommended that you leave > + this option empty or pick one of `committed`, `added`, > + or `all`. > ++ > +When this configuration is encountered, the set of components starts with > +the platform default value, disabled components are removed, and additional > +components are added. `none` resets the state so that the platform default > +is ignored. > ++ > +The empty string resets the fsync configuration to the platform > +default. The default on most platforms is equivalent to > +`core.fsync=committed,-loose-object`, which has good performance, > +but risks losing recent work in the event of an unclean system shutdown. > ++ > +* `none` clears the set of fsynced components. > +* `loose-object` hardens objects added to the repo in loose-object form. > +* `pack` hardens objects added to the repo in packfile form. > +* `pack-metadata` hardens packfile bitmaps and indexes. > +* `commit-graph` hardens the commit graph file. > +* `index` hardens the index when it is modified. > +* `objects` is an aggregate option that is equivalent to > + `loose-object,pack`. > +* `derived-metadata` is an aggregate option that is equivalent to > + `pack-metadata,commit-graph`. > +* `committed` is an aggregate option that is currently equivalent to > + `objects`. This mode sacrifices some performance to ensure that work > + that is committed to the repository with `git commit` or similar commands > + is hardened. > +* `added` is an aggregate option that is currently equivalent to > + `committed,index`. This mode sacrifices additional performance to > + ensure that the results of commands like `git add` and similar operations > + are hardened. > +* `all` is an aggregate option that syncs all individual components above. > + > core.fsyncMethod:: > A value indicating the strategy Git will use to harden repository data > using fsync and related primitives. On top of my https://lore.kernel.org/git/RFC-patch-v2-7.7-a5951366c6e-20220323T140753Z-avarab@gmail.com/ which makes the tmp-objdir part of your not-in-next-just-seen follow-up series configurable via "fsyncMethod.batch.quarantine" I really think we should just go for something like the belwo patch (note that misspelled/mistook "bulk" for "batch" in that linked-t patch, fixed below. I.e. I think we should just do our default fsync() of everything, and probably SOON make the fsync-ing of loose objects the default. Those who care about performance will have "batch" (or "writeout-only"), which we can have OS-specific detections for. But really, all of the rest of this is unduly boxing us into overconfiguration that I think nobody really needs. If someone really needs this level of detail they can LD_PRELOAD something to have fsync intercept fd's and paths, and act appropriately. Worse, as the RFC series I sent (https://lore.kernel.org/git/RFC-cover-v2-0.7-00000000000-20220323T140753Z-avarab@gmail.com/) shows we can and should "batch" up fsync() operations across these configuration boundaries, which this level of configuration would seem to preclude. Or, we'd need to explain why "core.fsync=loose-object" won't *actually* call fsync() on a single loose object's fd under "batch" as I had to do on top of this in https://lore.kernel.org/git/RFC-patch-v2-6.7-c20301d7967-20220323T140753Z-avarab@gmail.com/ The same is going to apply for almost all of the rest of these configuration categories. I.e. a natural follow-up to e.g. batching across objects & index as I'm doing in https://lore.kernel.org/git/RFC-patch-v2-4.7-61f4f3d7ef4-20220323T140753Z-avarab@gmail.com/ is to do likewise for all the PACK-related stuff before we rename it in-place. Or even have "git gc" issue only a single fsync() for all of PACKs, their metadata files, commit-graph etc., and then rename() things in-place as appropriate afterwards. diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt index 365a12dc7ae..536238e209b 100644 --- a/Documentation/config/core.txt +++ b/Documentation/config/core.txt @@ -548,49 +548,35 @@ core.whitespace:: errors. The default tab width is 8. Allowed values are 1 to 63. core.fsync:: - A comma-separated list of components of the repository that - should be hardened via the core.fsyncMethod when created or - modified. You can disable hardening of any component by - prefixing it with a '-'. Items that are not hardened may be - lost in the event of an unclean system shutdown. Unless you - have special requirements, it is recommended that you leave - this option empty or pick one of `committed`, `added`, - or `all`. -+ -When this configuration is encountered, the set of components starts with -the platform default value, disabled components are removed, and additional -components are added. `none` resets the state so that the platform default -is ignored. -+ -The empty string resets the fsync configuration to the platform -default. The default on most platforms is equivalent to -`core.fsync=committed,-loose-object`, which has good performance, -but risks losing recent work in the event of an unclean system shutdown. -+ -* `none` clears the set of fsynced components. -* `loose-object` hardens objects added to the repo in loose-object form. -* `pack` hardens objects added to the repo in packfile form. -* `pack-metadata` hardens packfile bitmaps and indexes. -* `commit-graph` hardens the commit graph file. -* `index` hardens the index when it is modified. -* `objects` is an aggregate option that is equivalent to - `loose-object,pack`. -* `derived-metadata` is an aggregate option that is equivalent to - `pack-metadata,commit-graph`. -* `committed` is an aggregate option that is currently equivalent to - `objects`. This mode sacrifices some performance to ensure that work - that is committed to the repository with `git commit` or similar commands - is hardened. -* `added` is an aggregate option that is currently equivalent to - `committed,index`. This mode sacrifices additional performance to - ensure that the results of commands like `git add` and similar operations - are hardened. -* `all` is an aggregate option that syncs all individual components above. + A boolen defaulting to `true`. To ensure data integrity git + will fsync() its objects, index and refu updates etc. This can + be set to `false` to disable `fsync()`-ing. ++ +Only set this to `false` if you know what you're doing, and are +prepared to deal with data corruption. Valid use-cases include +throwaway uses of repositories on ramdisks, one-off mass-imports +followed by calling `sync(1)` etc. ++ +Note that the syncing of loose objects is currently excluded from +`core.fsync=true`. To turn on all fsync-ing you'll need +`core.fsync=true` and `core.fsyncObjectFiles=true`, but see +`core.fsyncMethod=batch` below for a much faster alternative that's +just as safe on various modern OS's. ++ +The default is in flux and may change in the future, in particular the +equivalent of the already-deprecated `core.fsyncObjectFiles` setting +might soon default to `true`, and `core.fsyncMethod`'s default of +`fsync` might default to a setting deemed to be safe on the local OS, +suc has `batch` or `writeout-only` core.fsyncMethod:: A value indicating the strategy Git will use to harden repository data using fsync and related primitives. + +Defaults to `fsync`, but as discussed for `core.fsync` above might +change to use one of the values below taking advantage of +platform-specific "faster `fsync()`". ++ * `fsync` uses the fsync() system call or platform equivalents. * `writeout-only` issues pagecache writeback requests, but depending on the filesystem and storage hardware, data added to the repository may not be @@ -680,8 +666,8 @@ backed up by any standard (e.g. POSIX), but worked in practice on some Linux setups. + Nowadays you should almost certainly want to use -`core.fsync=loose-object` instead in combination with -`core.fsyncMethod=bulk`, and possibly with +`core.fsync=true` instead in combination with +`core.fsyncMethod=batch`, and possibly with `fsyncMethod.batch.quarantine=true`, see above. On modern OS's (Linux, OSX, Windows) that gives you most of the performance benefit of `core.fsyncObjectFiles=false` with all of the safety of the old ^ permalink raw reply related [flat|nested] 122+ messages in thread
* Re: do we have too much fsync() configuration in 'next'? (was: [PATCH v7] core.fsync: documentation and user-friendly aggregate options) 2022-03-23 14:20 ` do we have too much fsync() configuration in 'next'? (was: [PATCH v7] core.fsync: documentation and user-friendly aggregate options) Ævar Arnfjörð Bjarmason @ 2022-03-25 21:24 ` Neeraj Singh 2022-03-26 0:24 ` Ævar Arnfjörð Bjarmason 0 siblings, 1 reply; 122+ messages in thread From: Neeraj Singh @ 2022-03-25 21:24 UTC (permalink / raw) To: Ævar Arnfjörð Bjarmason Cc: Neeraj K. Singh via GitGitGadget, Johannes Schindelin, Bagas Sanjaya, Git List, Neeraj Singh, Elijah Newren, Patrick Steinhardt, Randall S. Becker, brian m. carlson On Wed, Mar 23, 2022 at 7:46 AM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote: > > > On Tue, Mar 15 2022, Neeraj Singh wrote: > > I know this is probably 80% my fault by egging you on about initially > adding the wildmatch() based thing you didn't go for. > > But having looked at this with fresh eyes quite deeply I really think > we're severely over-configuring things here: > > > +core.fsync:: > > + A comma-separated list of components of the repository that > > + should be hardened via the core.fsyncMethod when created or > > + modified. You can disable hardening of any component by > > + prefixing it with a '-'. Items that are not hardened may be > > + lost in the event of an unclean system shutdown. Unless you > > + have special requirements, it is recommended that you leave > > + this option empty or pick one of `committed`, `added`, > > + or `all`. > > ++ > > +When this configuration is encountered, the set of components starts with > > +the platform default value, disabled components are removed, and additional > > +components are added. `none` resets the state so that the platform default > > +is ignored. > > ++ > > +The empty string resets the fsync configuration to the platform > > +default. The default on most platforms is equivalent to > > +`core.fsync=committed,-loose-object`, which has good performance, > > +but risks losing recent work in the event of an unclean system shutdown. > > ++ > > +* `none` clears the set of fsynced components. > > +* `loose-object` hardens objects added to the repo in loose-object form. > > +* `pack` hardens objects added to the repo in packfile form. > > +* `pack-metadata` hardens packfile bitmaps and indexes. > > +* `commit-graph` hardens the commit graph file. > > +* `index` hardens the index when it is modified. > > +* `objects` is an aggregate option that is equivalent to > > + `loose-object,pack`. > > +* `derived-metadata` is an aggregate option that is equivalent to > > + `pack-metadata,commit-graph`. > > +* `committed` is an aggregate option that is currently equivalent to > > + `objects`. This mode sacrifices some performance to ensure that work > > + that is committed to the repository with `git commit` or similar commands > > + is hardened. > > +* `added` is an aggregate option that is currently equivalent to > > + `committed,index`. This mode sacrifices additional performance to > > + ensure that the results of commands like `git add` and similar operations > > + are hardened. > > +* `all` is an aggregate option that syncs all individual components above. > > + > > core.fsyncMethod:: > > A value indicating the strategy Git will use to harden repository data > > using fsync and related primitives. > > On top of my > https://lore.kernel.org/git/RFC-patch-v2-7.7-a5951366c6e-20220323T140753Z-avarab@gmail.com/ > which makes the tmp-objdir part of your not-in-next-just-seen follow-up > series configurable via "fsyncMethod.batch.quarantine" I really think we > should just go for something like the belwo patch (note that > misspelled/mistook "bulk" for "batch" in that linked-t patch, fixed > below. > > I.e. I think we should just do our default fsync() of everything, and > probably SOON make the fsync-ing of loose objects the default. Those who > care about performance will have "batch" (or "writeout-only"), which we > can have OS-specific detections for. > > But really, all of the rest of this is unduly boxing us into > overconfiguration that I think nobody really needs. > We've gone over this a few times already, but just wanted to state it again. The really detailed settings are really there for Git hosters like GitLab or GitHub. I'd be happy to remove the per-component core.fsync values from the documentation and leave just the ones we point the user to. > If someone really needs this level of detail they can LD_PRELOAD > something to have fsync intercept fd's and paths, and act appropriately. > > Worse, as the RFC series I sent > (https://lore.kernel.org/git/RFC-cover-v2-0.7-00000000000-20220323T140753Z-avarab@gmail.com/) > shows we can and should "batch" up fsync() operations across these > configuration boundaries, which this level of configuration would seem > to preclude. > > Or, we'd need to explain why "core.fsync=loose-object" won't *actually* > call fsync() on a single loose object's fd under "batch" as I had to do > on top of this in > https://lore.kernel.org/git/RFC-patch-v2-6.7-c20301d7967-20220323T140753Z-avarab@gmail.com/ > 99.9% of users don't care and won't look. The ones who do look deeper and understand the issues have source code and access to this ML discussion to understand why this works this way. > The same is going to apply for almost all of the rest of these > configuration categories. > > I.e. a natural follow-up to e.g. batching across objects & index as I'm > doing in > https://lore.kernel.org/git/RFC-patch-v2-4.7-61f4f3d7ef4-20220323T140753Z-avarab@gmail.com/ > is to do likewise for all the PACK-related stuff before we rename it > in-place. Or even have "git gc" issue only a single fsync() for all of > PACKs, their metadata files, commit-graph etc., and then rename() things > in-place as appropriate afterwards. > > diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt > index 365a12dc7ae..536238e209b 100644 > --- a/Documentation/config/core.txt > +++ b/Documentation/config/core.txt > @@ -548,49 +548,35 @@ core.whitespace:: > errors. The default tab width is 8. Allowed values are 1 to 63. > > core.fsync:: > - A comma-separated list of components of the repository that > - should be hardened via the core.fsyncMethod when created or > - modified. You can disable hardening of any component by > - prefixing it with a '-'. Items that are not hardened may be > - lost in the event of an unclean system shutdown. Unless you > - have special requirements, it is recommended that you leave > - this option empty or pick one of `committed`, `added`, > - or `all`. > -+ > -When this configuration is encountered, the set of components starts with > -the platform default value, disabled components are removed, and additional > -components are added. `none` resets the state so that the platform default > -is ignored. > -+ > -The empty string resets the fsync configuration to the platform > -default. The default on most platforms is equivalent to > -`core.fsync=committed,-loose-object`, which has good performance, > -but risks losing recent work in the event of an unclean system shutdown. > -+ > -* `none` clears the set of fsynced components. > -* `loose-object` hardens objects added to the repo in loose-object form. > -* `pack` hardens objects added to the repo in packfile form. > -* `pack-metadata` hardens packfile bitmaps and indexes. > -* `commit-graph` hardens the commit graph file. > -* `index` hardens the index when it is modified. > -* `objects` is an aggregate option that is equivalent to > - `loose-object,pack`. > -* `derived-metadata` is an aggregate option that is equivalent to > - `pack-metadata,commit-graph`. > -* `committed` is an aggregate option that is currently equivalent to > - `objects`. This mode sacrifices some performance to ensure that work > - that is committed to the repository with `git commit` or similar commands > - is hardened. > -* `added` is an aggregate option that is currently equivalent to > - `committed,index`. This mode sacrifices additional performance to > - ensure that the results of commands like `git add` and similar operations > - are hardened. > -* `all` is an aggregate option that syncs all individual components above. > + A boolen defaulting to `true`. To ensure data integrity git > + will fsync() its objects, index and refu updates etc. This can > + be set to `false` to disable `fsync()`-ing. > ++ > +Only set this to `false` if you know what you're doing, and are > +prepared to deal with data corruption. Valid use-cases include > +throwaway uses of repositories on ramdisks, one-off mass-imports > +followed by calling `sync(1)` etc. > ++ > +Note that the syncing of loose objects is currently excluded from > +`core.fsync=true`. To turn on all fsync-ing you'll need > +`core.fsync=true` and `core.fsyncObjectFiles=true`, but see > +`core.fsyncMethod=batch` below for a much faster alternative that's > +just as safe on various modern OS's. > ++ > +The default is in flux and may change in the future, in particular the > +equivalent of the already-deprecated `core.fsyncObjectFiles` setting > +might soon default to `true`, and `core.fsyncMethod`'s default of > +`fsync` might default to a setting deemed to be safe on the local OS, > +suc has `batch` or `writeout-only` > > core.fsyncMethod:: > A value indicating the strategy Git will use to harden repository data > using fsync and related primitives. > + > +Defaults to `fsync`, but as discussed for `core.fsync` above might > +change to use one of the values below taking advantage of > +platform-specific "faster `fsync()`". > ++ > * `fsync` uses the fsync() system call or platform equivalents. > * `writeout-only` issues pagecache writeback requests, but depending on the > filesystem and storage hardware, data added to the repository may not be > @@ -680,8 +666,8 @@ backed up by any standard (e.g. POSIX), but worked in practice on some > Linux setups. > + > Nowadays you should almost certainly want to use > -`core.fsync=loose-object` instead in combination with > -`core.fsyncMethod=bulk`, and possibly with > +`core.fsync=true` instead in combination with > +`core.fsyncMethod=batch`, and possibly with > `fsyncMethod.batch.quarantine=true`, see above. On modern OS's (Linux, > OSX, Windows) that gives you most of the performance benefit of > `core.fsyncObjectFiles=false` with all of the safety of the old I'm at the point where I don't want to endlessly revisit this discussion. -Neeraj ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: do we have too much fsync() configuration in 'next'? (was: [PATCH v7] core.fsync: documentation and user-friendly aggregate options) 2022-03-25 21:24 ` Neeraj Singh @ 2022-03-26 0:24 ` Ævar Arnfjörð Bjarmason 2022-03-26 1:23 ` do we have too much fsync() configuration in 'next'? Junio C Hamano 2022-03-26 1:25 ` do we have too much fsync() configuration in 'next'? (was: [PATCH v7] core.fsync: documentation and user-friendly aggregate options) Neeraj Singh 0 siblings, 2 replies; 122+ messages in thread From: Ævar Arnfjörð Bjarmason @ 2022-03-26 0:24 UTC (permalink / raw) To: Neeraj Singh Cc: Neeraj K. Singh via GitGitGadget, Johannes Schindelin, Bagas Sanjaya, Git List, Neeraj Singh, Elijah Newren, Patrick Steinhardt, Randall S. Becker, brian m. carlson On Fri, Mar 25 2022, Neeraj Singh wrote: > On Wed, Mar 23, 2022 at 7:46 AM Ævar Arnfjörð Bjarmason > <avarab@gmail.com> wrote: >> >> >> On Tue, Mar 15 2022, Neeraj Singh wrote: >> >> I know this is probably 80% my fault by egging you on about initially >> adding the wildmatch() based thing you didn't go for. >> >> But having looked at this with fresh eyes quite deeply I really think >> we're severely over-configuring things here: >> >> > +core.fsync:: >> > + A comma-separated list of components of the repository that >> > + should be hardened via the core.fsyncMethod when created or >> > + modified. You can disable hardening of any component by >> > + prefixing it with a '-'. Items that are not hardened may be >> > + lost in the event of an unclean system shutdown. Unless you >> > + have special requirements, it is recommended that you leave >> > + this option empty or pick one of `committed`, `added`, >> > + or `all`. >> > ++ >> > +When this configuration is encountered, the set of components starts with >> > +the platform default value, disabled components are removed, and additional >> > +components are added. `none` resets the state so that the platform default >> > +is ignored. >> > ++ >> > +The empty string resets the fsync configuration to the platform >> > +default. The default on most platforms is equivalent to >> > +`core.fsync=committed,-loose-object`, which has good performance, >> > +but risks losing recent work in the event of an unclean system shutdown. >> > ++ >> > +* `none` clears the set of fsynced components. >> > +* `loose-object` hardens objects added to the repo in loose-object form. >> > +* `pack` hardens objects added to the repo in packfile form. >> > +* `pack-metadata` hardens packfile bitmaps and indexes. >> > +* `commit-graph` hardens the commit graph file. >> > +* `index` hardens the index when it is modified. >> > +* `objects` is an aggregate option that is equivalent to >> > + `loose-object,pack`. >> > +* `derived-metadata` is an aggregate option that is equivalent to >> > + `pack-metadata,commit-graph`. >> > +* `committed` is an aggregate option that is currently equivalent to >> > + `objects`. This mode sacrifices some performance to ensure that work >> > + that is committed to the repository with `git commit` or similar commands >> > + is hardened. >> > +* `added` is an aggregate option that is currently equivalent to >> > + `committed,index`. This mode sacrifices additional performance to >> > + ensure that the results of commands like `git add` and similar operations >> > + are hardened. >> > +* `all` is an aggregate option that syncs all individual components above. >> > + >> > core.fsyncMethod:: >> > A value indicating the strategy Git will use to harden repository data >> > using fsync and related primitives. >> >> On top of my >> https://lore.kernel.org/git/RFC-patch-v2-7.7-a5951366c6e-20220323T140753Z-avarab@gmail.com/ >> which makes the tmp-objdir part of your not-in-next-just-seen follow-up >> series configurable via "fsyncMethod.batch.quarantine" I really think we >> should just go for something like the belwo patch (note that >> misspelled/mistook "bulk" for "batch" in that linked-t patch, fixed >> below. >> >> I.e. I think we should just do our default fsync() of everything, and >> probably SOON make the fsync-ing of loose objects the default. Those who >> care about performance will have "batch" (or "writeout-only"), which we >> can have OS-specific detections for. >> >> But really, all of the rest of this is unduly boxing us into >> overconfiguration that I think nobody really needs. >> > > We've gone over this a few times already, but just wanted to state it > again. The really detailed settings are really there for Git hosters > like GitLab or GitHub. I'd be happy to remove the per-component > core.fsync values from the documentation and leave just the ones we > point the user to. I'm prettty sure (but Patrick knows more) that GitLab's plan for this is to keep it at whatever the safest setting is, presumably GitHub's as well (but I don't know at all on that front). >> If someone really needs this level of detail they can LD_PRELOAD >> something to have fsync intercept fd's and paths, and act appropriately. >> >> Worse, as the RFC series I sent >> (https://lore.kernel.org/git/RFC-cover-v2-0.7-00000000000-20220323T140753Z-avarab@gmail.com/) >> shows we can and should "batch" up fsync() operations across these >> configuration boundaries, which this level of configuration would seem >> to preclude. >> >> Or, we'd need to explain why "core.fsync=loose-object" won't *actually* >> call fsync() on a single loose object's fd under "batch" as I had to do >> on top of this in >> https://lore.kernel.org/git/RFC-patch-v2-6.7-c20301d7967-20220323T140753Z-avarab@gmail.com/ >> > > 99.9% of users don't care and won't look. The ones who do look deeper > and understand the issues have source code and access to this ML > discussion to understand why this works this way. Exactly, so we can hopefully have a simpler interface. >> The same is going to apply for almost all of the rest of these >> configuration categories. >> >> I.e. a natural follow-up to e.g. batching across objects & index as I'm >> doing in >> https://lore.kernel.org/git/RFC-patch-v2-4.7-61f4f3d7ef4-20220323T140753Z-avarab@gmail.com/ >> is to do likewise for all the PACK-related stuff before we rename it >> in-place. Or even have "git gc" issue only a single fsync() for all of >> PACKs, their metadata files, commit-graph etc., and then rename() things >> in-place as appropriate afterwards. >> >> diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt >> index 365a12dc7ae..536238e209b 100644 >> --- a/Documentation/config/core.txt >> +++ b/Documentation/config/core.txt >> @@ -548,49 +548,35 @@ core.whitespace:: >> errors. The default tab width is 8. Allowed values are 1 to 63. >> >> core.fsync:: >> - A comma-separated list of components of the repository that >> - should be hardened via the core.fsyncMethod when created or >> - modified. You can disable hardening of any component by >> - prefixing it with a '-'. Items that are not hardened may be >> - lost in the event of an unclean system shutdown. Unless you >> - have special requirements, it is recommended that you leave >> - this option empty or pick one of `committed`, `added`, >> - or `all`. >> -+ >> -When this configuration is encountered, the set of components starts with >> -the platform default value, disabled components are removed, and additional >> -components are added. `none` resets the state so that the platform default >> -is ignored. >> -+ >> -The empty string resets the fsync configuration to the platform >> -default. The default on most platforms is equivalent to >> -`core.fsync=committed,-loose-object`, which has good performance, >> -but risks losing recent work in the event of an unclean system shutdown. >> -+ >> -* `none` clears the set of fsynced components. >> -* `loose-object` hardens objects added to the repo in loose-object form. >> -* `pack` hardens objects added to the repo in packfile form. >> -* `pack-metadata` hardens packfile bitmaps and indexes. >> -* `commit-graph` hardens the commit graph file. >> -* `index` hardens the index when it is modified. >> -* `objects` is an aggregate option that is equivalent to >> - `loose-object,pack`. >> -* `derived-metadata` is an aggregate option that is equivalent to >> - `pack-metadata,commit-graph`. >> -* `committed` is an aggregate option that is currently equivalent to >> - `objects`. This mode sacrifices some performance to ensure that work >> - that is committed to the repository with `git commit` or similar commands >> - is hardened. >> -* `added` is an aggregate option that is currently equivalent to >> - `committed,index`. This mode sacrifices additional performance to >> - ensure that the results of commands like `git add` and similar operations >> - are hardened. >> -* `all` is an aggregate option that syncs all individual components above. >> + A boolen defaulting to `true`. To ensure data integrity git >> + will fsync() its objects, index and refu updates etc. This can >> + be set to `false` to disable `fsync()`-ing. >> ++ >> +Only set this to `false` if you know what you're doing, and are >> +prepared to deal with data corruption. Valid use-cases include >> +throwaway uses of repositories on ramdisks, one-off mass-imports >> +followed by calling `sync(1)` etc. >> ++ >> +Note that the syncing of loose objects is currently excluded from >> +`core.fsync=true`. To turn on all fsync-ing you'll need >> +`core.fsync=true` and `core.fsyncObjectFiles=true`, but see >> +`core.fsyncMethod=batch` below for a much faster alternative that's >> +just as safe on various modern OS's. >> ++ >> +The default is in flux and may change in the future, in particular the >> +equivalent of the already-deprecated `core.fsyncObjectFiles` setting >> +might soon default to `true`, and `core.fsyncMethod`'s default of >> +`fsync` might default to a setting deemed to be safe on the local OS, >> +suc has `batch` or `writeout-only` >> >> core.fsyncMethod:: >> A value indicating the strategy Git will use to harden repository data >> using fsync and related primitives. >> + >> +Defaults to `fsync`, but as discussed for `core.fsync` above might >> +change to use one of the values below taking advantage of >> +platform-specific "faster `fsync()`". >> ++ >> * `fsync` uses the fsync() system call or platform equivalents. >> * `writeout-only` issues pagecache writeback requests, but depending on the >> filesystem and storage hardware, data added to the repository may not be >> @@ -680,8 +666,8 @@ backed up by any standard (e.g. POSIX), but worked in practice on some >> Linux setups. >> + >> Nowadays you should almost certainly want to use >> -`core.fsync=loose-object` instead in combination with >> -`core.fsyncMethod=bulk`, and possibly with >> +`core.fsync=true` instead in combination with >> +`core.fsyncMethod=batch`, and possibly with >> `fsyncMethod.batch.quarantine=true`, see above. On modern OS's (Linux, >> OSX, Windows) that gives you most of the performance benefit of >> `core.fsyncObjectFiles=false` with all of the safety of the old > > I'm at the point where I don't want to endlessly revisit this discussion. Sorry, my intention isn't to frustrate you, but I do think it's important to get this right. Particularly since this is now in "next", and we're getting closer to a release. We can either talk about this now and decide on something, or it'll be in a release, and then publicly documented promises will be harder to back out of. I think your suggestion of just hiding the relevant documentation would be a good band-aid solution to that. But I also think that given how I was altering this in my RFC series that the premise of how this could be structured has been called into question in a way that we didn't (or I don't recall) us having discussed before. I.e. that we can say "sync loose, but not index", or "sync index, but not loose" with this config schema. When with "bulk" we it really isn't any more expensive to do both if one is true (even cheaper, actually). ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: do we have too much fsync() configuration in 'next'? 2022-03-26 0:24 ` Ævar Arnfjörð Bjarmason @ 2022-03-26 1:23 ` Junio C Hamano 2022-03-26 1:25 ` do we have too much fsync() configuration in 'next'? (was: [PATCH v7] core.fsync: documentation and user-friendly aggregate options) Neeraj Singh 1 sibling, 0 replies; 122+ messages in thread From: Junio C Hamano @ 2022-03-26 1:23 UTC (permalink / raw) To: Ævar Arnfjörð Bjarmason Cc: Neeraj Singh, Neeraj K. Singh via GitGitGadget, Johannes Schindelin, Bagas Sanjaya, Git List, Neeraj Singh, Elijah Newren, Patrick Steinhardt, Randall S. Becker, brian m. carlson Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes: >> We've gone over this a few times already, but just wanted to state it >> again. The really detailed settings are really there for Git hosters >> like GitLab or GitHub. I'd be happy to remove the per-component >> core.fsync values from the documentation and leave just the ones we >> point the user to. > > I'm prettty sure (but Patrick knows more) that GitLab's plan for this is > to keep it at whatever the safest setting is, presumably GitHub's as > well (but I don't know at all on that front). I thought we've already settled it long ago, and it wasn't like you were taking vacation from the list for a few weeks. Why are we talking about this again now? Did we discover any new argument against it? Puzzled. ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: do we have too much fsync() configuration in 'next'? (was: [PATCH v7] core.fsync: documentation and user-friendly aggregate options) 2022-03-26 0:24 ` Ævar Arnfjörð Bjarmason 2022-03-26 1:23 ` do we have too much fsync() configuration in 'next'? Junio C Hamano @ 2022-03-26 1:25 ` Neeraj Singh 2022-03-26 15:31 ` Ævar Arnfjörð Bjarmason 1 sibling, 1 reply; 122+ messages in thread From: Neeraj Singh @ 2022-03-26 1:25 UTC (permalink / raw) To: Ævar Arnfjörð Bjarmason Cc: Neeraj K. Singh via GitGitGadget, Johannes Schindelin, Bagas Sanjaya, Git List, Neeraj Singh, Elijah Newren, Patrick Steinhardt, Randall S. Becker, brian m. carlson On Fri, Mar 25, 2022 at 5:33 PM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote: > > > On Fri, Mar 25 2022, Neeraj Singh wrote: > > > On Wed, Mar 23, 2022 at 7:46 AM Ævar Arnfjörð Bjarmason > > <avarab@gmail.com> wrote: > >> > >> > >> On Tue, Mar 15 2022, Neeraj Singh wrote: > >> > >> I know this is probably 80% my fault by egging you on about initially > >> adding the wildmatch() based thing you didn't go for. > >> > >> But having looked at this with fresh eyes quite deeply I really think > >> we're severely over-configuring things here: > >> > >> > +core.fsync:: > >> > + A comma-separated list of components of the repository that > >> > + should be hardened via the core.fsyncMethod when created or > >> > + modified. You can disable hardening of any component by > >> > + prefixing it with a '-'. Items that are not hardened may be > >> > + lost in the event of an unclean system shutdown. Unless you > >> > + have special requirements, it is recommended that you leave > >> > + this option empty or pick one of `committed`, `added`, > >> > + or `all`. > >> > ++ > >> > +When this configuration is encountered, the set of components starts with > >> > +the platform default value, disabled components are removed, and additional > >> > +components are added. `none` resets the state so that the platform default > >> > +is ignored. > >> > ++ > >> > +The empty string resets the fsync configuration to the platform > >> > +default. The default on most platforms is equivalent to > >> > +`core.fsync=committed,-loose-object`, which has good performance, > >> > +but risks losing recent work in the event of an unclean system shutdown. > >> > ++ > >> > +* `none` clears the set of fsynced components. > >> > +* `loose-object` hardens objects added to the repo in loose-object form. > >> > +* `pack` hardens objects added to the repo in packfile form. > >> > +* `pack-metadata` hardens packfile bitmaps and indexes. > >> > +* `commit-graph` hardens the commit graph file. > >> > +* `index` hardens the index when it is modified. > >> > +* `objects` is an aggregate option that is equivalent to > >> > + `loose-object,pack`. > >> > +* `derived-metadata` is an aggregate option that is equivalent to > >> > + `pack-metadata,commit-graph`. > >> > +* `committed` is an aggregate option that is currently equivalent to > >> > + `objects`. This mode sacrifices some performance to ensure that work > >> > + that is committed to the repository with `git commit` or similar commands > >> > + is hardened. > >> > +* `added` is an aggregate option that is currently equivalent to > >> > + `committed,index`. This mode sacrifices additional performance to > >> > + ensure that the results of commands like `git add` and similar operations > >> > + are hardened. > >> > +* `all` is an aggregate option that syncs all individual components above. > >> > + > >> > core.fsyncMethod:: > >> > A value indicating the strategy Git will use to harden repository data > >> > using fsync and related primitives. > >> > >> On top of my > >> https://lore.kernel.org/git/RFC-patch-v2-7.7-a5951366c6e-20220323T140753Z-avarab@gmail.com/ > >> which makes the tmp-objdir part of your not-in-next-just-seen follow-up > >> series configurable via "fsyncMethod.batch.quarantine" I really think we > >> should just go for something like the belwo patch (note that > >> misspelled/mistook "bulk" for "batch" in that linked-t patch, fixed > >> below. > >> > >> I.e. I think we should just do our default fsync() of everything, and > >> probably SOON make the fsync-ing of loose objects the default. Those who > >> care about performance will have "batch" (or "writeout-only"), which we > >> can have OS-specific detections for. > >> > >> But really, all of the rest of this is unduly boxing us into > >> overconfiguration that I think nobody really needs. > >> > > > > We've gone over this a few times already, but just wanted to state it > > again. The really detailed settings are really there for Git hosters > > like GitLab or GitHub. I'd be happy to remove the per-component > > core.fsync values from the documentation and leave just the ones we > > point the user to. > > I'm prettty sure (but Patrick knows more) that GitLab's plan for this is > to keep it at whatever the safest setting is, presumably GitHub's as > well (but I don't know at all on that front). > > >> If someone really needs this level of detail they can LD_PRELOAD > >> something to have fsync intercept fd's and paths, and act appropriately. > >> > >> Worse, as the RFC series I sent > >> (https://lore.kernel.org/git/RFC-cover-v2-0.7-00000000000-20220323T140753Z-avarab@gmail.com/) > >> shows we can and should "batch" up fsync() operations across these > >> configuration boundaries, which this level of configuration would seem > >> to preclude. > >> > >> Or, we'd need to explain why "core.fsync=loose-object" won't *actually* > >> call fsync() on a single loose object's fd under "batch" as I had to do > >> on top of this in > >> https://lore.kernel.org/git/RFC-patch-v2-6.7-c20301d7967-20220323T140753Z-avarab@gmail.com/ > >> > > > > 99.9% of users don't care and won't look. The ones who do look deeper > > and understand the issues have source code and access to this ML > > discussion to understand why this works this way. > > Exactly, so we can hopefully have a simpler interface. > > >> The same is going to apply for almost all of the rest of these > >> configuration categories. > >> > >> I.e. a natural follow-up to e.g. batching across objects & index as I'm > >> doing in > >> https://lore.kernel.org/git/RFC-patch-v2-4.7-61f4f3d7ef4-20220323T140753Z-avarab@gmail.com/ > >> is to do likewise for all the PACK-related stuff before we rename it > >> in-place. Or even have "git gc" issue only a single fsync() for all of > >> PACKs, their metadata files, commit-graph etc., and then rename() things > >> in-place as appropriate afterwards. > >> > >> diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt > >> index 365a12dc7ae..536238e209b 100644 > >> --- a/Documentation/config/core.txt > >> +++ b/Documentation/config/core.txt > >> @@ -548,49 +548,35 @@ core.whitespace:: > >> errors. The default tab width is 8. Allowed values are 1 to 63. > >> > >> core.fsync:: > >> - A comma-separated list of components of the repository that > >> - should be hardened via the core.fsyncMethod when created or > >> - modified. You can disable hardening of any component by > >> - prefixing it with a '-'. Items that are not hardened may be > >> - lost in the event of an unclean system shutdown. Unless you > >> - have special requirements, it is recommended that you leave > >> - this option empty or pick one of `committed`, `added`, > >> - or `all`. > >> -+ > >> -When this configuration is encountered, the set of components starts with > >> -the platform default value, disabled components are removed, and additional > >> -components are added. `none` resets the state so that the platform default > >> -is ignored. > >> -+ > >> -The empty string resets the fsync configuration to the platform > >> -default. The default on most platforms is equivalent to > >> -`core.fsync=committed,-loose-object`, which has good performance, > >> -but risks losing recent work in the event of an unclean system shutdown. > >> -+ > >> -* `none` clears the set of fsynced components. > >> -* `loose-object` hardens objects added to the repo in loose-object form. > >> -* `pack` hardens objects added to the repo in packfile form. > >> -* `pack-metadata` hardens packfile bitmaps and indexes. > >> -* `commit-graph` hardens the commit graph file. > >> -* `index` hardens the index when it is modified. > >> -* `objects` is an aggregate option that is equivalent to > >> - `loose-object,pack`. > >> -* `derived-metadata` is an aggregate option that is equivalent to > >> - `pack-metadata,commit-graph`. > >> -* `committed` is an aggregate option that is currently equivalent to > >> - `objects`. This mode sacrifices some performance to ensure that work > >> - that is committed to the repository with `git commit` or similar commands > >> - is hardened. > >> -* `added` is an aggregate option that is currently equivalent to > >> - `committed,index`. This mode sacrifices additional performance to > >> - ensure that the results of commands like `git add` and similar operations > >> - are hardened. > >> -* `all` is an aggregate option that syncs all individual components above. > >> + A boolen defaulting to `true`. To ensure data integrity git > >> + will fsync() its objects, index and refu updates etc. This can > >> + be set to `false` to disable `fsync()`-ing. > >> ++ > >> +Only set this to `false` if you know what you're doing, and are > >> +prepared to deal with data corruption. Valid use-cases include > >> +throwaway uses of repositories on ramdisks, one-off mass-imports > >> +followed by calling `sync(1)` etc. > >> ++ > >> +Note that the syncing of loose objects is currently excluded from > >> +`core.fsync=true`. To turn on all fsync-ing you'll need > >> +`core.fsync=true` and `core.fsyncObjectFiles=true`, but see > >> +`core.fsyncMethod=batch` below for a much faster alternative that's > >> +just as safe on various modern OS's. > >> ++ > >> +The default is in flux and may change in the future, in particular the > >> +equivalent of the already-deprecated `core.fsyncObjectFiles` setting > >> +might soon default to `true`, and `core.fsyncMethod`'s default of > >> +`fsync` might default to a setting deemed to be safe on the local OS, > >> +suc has `batch` or `writeout-only` > >> > >> core.fsyncMethod:: > >> A value indicating the strategy Git will use to harden repository data > >> using fsync and related primitives. > >> + > >> +Defaults to `fsync`, but as discussed for `core.fsync` above might > >> +change to use one of the values below taking advantage of > >> +platform-specific "faster `fsync()`". > >> ++ > >> * `fsync` uses the fsync() system call or platform equivalents. > >> * `writeout-only` issues pagecache writeback requests, but depending on the > >> filesystem and storage hardware, data added to the repository may not be > >> @@ -680,8 +666,8 @@ backed up by any standard (e.g. POSIX), but worked in practice on some > >> Linux setups. > >> + > >> Nowadays you should almost certainly want to use > >> -`core.fsync=loose-object` instead in combination with > >> -`core.fsyncMethod=bulk`, and possibly with > >> +`core.fsync=true` instead in combination with > >> +`core.fsyncMethod=batch`, and possibly with > >> `fsyncMethod.batch.quarantine=true`, see above. On modern OS's (Linux, > >> OSX, Windows) that gives you most of the performance benefit of > >> `core.fsyncObjectFiles=false` with all of the safety of the old > > > > I'm at the point where I don't want to endlessly revisit this discussion. > > Sorry, my intention isn't to frustrate you, but I do think it's > important to get this right. > > Particularly since this is now in "next", and we're getting closer to a > release. We can either talk about this now and decide on something, or > it'll be in a release, and then publicly documented promises will be > harder to back out of. > > I think your suggestion of just hiding the relevant documentation would > be a good band-aid solution to that. > > But I also think that given how I was altering this in my RFC series > that the premise of how this could be structured has been called into > question in a way that we didn't (or I don't recall) us having discussed > before. > > I.e. that we can say "sync loose, but not index", or "sync index, but > not loose" with this config schema. When with "bulk" we it really isn't > any more expensive to do both if one is true (even cheaper, actually). > I want to make a comment about the Index here. Syncing the index is strictly required for the "added" level of consistency, so that we don't lose stuff that leaves the work tree but is staged. But my Windows enlistment has an index that's 266MB, which would be painful to sync even with all the optimizations. Maybe with split-index, this wouldn't be so bad, but I just wanted to call out that some advanced users may really care about the configurability. As Git's various database implementations improve, the fsync stuff will hopefully be more optimal and self-tuning. But as that happens, Git could just start ignoring settings that lose meaning without tying anyones hands. ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: do we have too much fsync() configuration in 'next'? (was: [PATCH v7] core.fsync: documentation and user-friendly aggregate options) 2022-03-26 1:25 ` do we have too much fsync() configuration in 'next'? (was: [PATCH v7] core.fsync: documentation and user-friendly aggregate options) Neeraj Singh @ 2022-03-26 15:31 ` Ævar Arnfjörð Bjarmason 2022-03-27 5:27 ` Neeraj Singh 0 siblings, 1 reply; 122+ messages in thread From: Ævar Arnfjörð Bjarmason @ 2022-03-26 15:31 UTC (permalink / raw) To: Neeraj Singh Cc: Neeraj K. Singh via GitGitGadget, Johannes Schindelin, Bagas Sanjaya, Git List, Neeraj Singh, Elijah Newren, Patrick Steinhardt, Randall S. Becker, brian m. carlson On Fri, Mar 25 2022, Neeraj Singh wrote: > On Fri, Mar 25, 2022 at 5:33 PM Ævar Arnfjörð Bjarmason > <avarab@gmail.com> wrote: >> >> >> On Fri, Mar 25 2022, Neeraj Singh wrote: >> >> > On Wed, Mar 23, 2022 at 7:46 AM Ævar Arnfjörð Bjarmason >> > <avarab@gmail.com> wrote: >> >> >> >> >> >> On Tue, Mar 15 2022, Neeraj Singh wrote: >> >> >> >> I know this is probably 80% my fault by egging you on about initially >> >> adding the wildmatch() based thing you didn't go for. >> >> >> >> But having looked at this with fresh eyes quite deeply I really think >> >> we're severely over-configuring things here: >> >> >> >> > +core.fsync:: >> >> > + A comma-separated list of components of the repository that >> >> > + should be hardened via the core.fsyncMethod when created or >> >> > + modified. You can disable hardening of any component by >> >> > + prefixing it with a '-'. Items that are not hardened may be >> >> > + lost in the event of an unclean system shutdown. Unless you >> >> > + have special requirements, it is recommended that you leave >> >> > + this option empty or pick one of `committed`, `added`, >> >> > + or `all`. >> >> > ++ >> >> > +When this configuration is encountered, the set of components starts with >> >> > +the platform default value, disabled components are removed, and additional >> >> > +components are added. `none` resets the state so that the platform default >> >> > +is ignored. >> >> > ++ >> >> > +The empty string resets the fsync configuration to the platform >> >> > +default. The default on most platforms is equivalent to >> >> > +`core.fsync=committed,-loose-object`, which has good performance, >> >> > +but risks losing recent work in the event of an unclean system shutdown. >> >> > ++ >> >> > +* `none` clears the set of fsynced components. >> >> > +* `loose-object` hardens objects added to the repo in loose-object form. >> >> > +* `pack` hardens objects added to the repo in packfile form. >> >> > +* `pack-metadata` hardens packfile bitmaps and indexes. >> >> > +* `commit-graph` hardens the commit graph file. >> >> > +* `index` hardens the index when it is modified. >> >> > +* `objects` is an aggregate option that is equivalent to >> >> > + `loose-object,pack`. >> >> > +* `derived-metadata` is an aggregate option that is equivalent to >> >> > + `pack-metadata,commit-graph`. >> >> > +* `committed` is an aggregate option that is currently equivalent to >> >> > + `objects`. This mode sacrifices some performance to ensure that work >> >> > + that is committed to the repository with `git commit` or similar commands >> >> > + is hardened. >> >> > +* `added` is an aggregate option that is currently equivalent to >> >> > + `committed,index`. This mode sacrifices additional performance to >> >> > + ensure that the results of commands like `git add` and similar operations >> >> > + are hardened. >> >> > +* `all` is an aggregate option that syncs all individual components above. >> >> > + >> >> > core.fsyncMethod:: >> >> > A value indicating the strategy Git will use to harden repository data >> >> > using fsync and related primitives. >> >> >> >> On top of my >> >> https://lore.kernel.org/git/RFC-patch-v2-7.7-a5951366c6e-20220323T140753Z-avarab@gmail.com/ >> >> which makes the tmp-objdir part of your not-in-next-just-seen follow-up >> >> series configurable via "fsyncMethod.batch.quarantine" I really think we >> >> should just go for something like the belwo patch (note that >> >> misspelled/mistook "bulk" for "batch" in that linked-t patch, fixed >> >> below. >> >> >> >> I.e. I think we should just do our default fsync() of everything, and >> >> probably SOON make the fsync-ing of loose objects the default. Those who >> >> care about performance will have "batch" (or "writeout-only"), which we >> >> can have OS-specific detections for. >> >> >> >> But really, all of the rest of this is unduly boxing us into >> >> overconfiguration that I think nobody really needs. >> >> >> > >> > We've gone over this a few times already, but just wanted to state it >> > again. The really detailed settings are really there for Git hosters >> > like GitLab or GitHub. I'd be happy to remove the per-component >> > core.fsync values from the documentation and leave just the ones we >> > point the user to. >> >> I'm prettty sure (but Patrick knows more) that GitLab's plan for this is >> to keep it at whatever the safest setting is, presumably GitHub's as >> well (but I don't know at all on that front). >> >> >> If someone really needs this level of detail they can LD_PRELOAD >> >> something to have fsync intercept fd's and paths, and act appropriately. >> >> >> >> Worse, as the RFC series I sent >> >> (https://lore.kernel.org/git/RFC-cover-v2-0.7-00000000000-20220323T140753Z-avarab@gmail.com/) >> >> shows we can and should "batch" up fsync() operations across these >> >> configuration boundaries, which this level of configuration would seem >> >> to preclude. >> >> >> >> Or, we'd need to explain why "core.fsync=loose-object" won't *actually* >> >> call fsync() on a single loose object's fd under "batch" as I had to do >> >> on top of this in >> >> https://lore.kernel.org/git/RFC-patch-v2-6.7-c20301d7967-20220323T140753Z-avarab@gmail.com/ >> >> >> > >> > 99.9% of users don't care and won't look. The ones who do look deeper >> > and understand the issues have source code and access to this ML >> > discussion to understand why this works this way. >> >> Exactly, so we can hopefully have a simpler interface. >> >> >> The same is going to apply for almost all of the rest of these >> >> configuration categories. >> >> >> >> I.e. a natural follow-up to e.g. batching across objects & index as I'm >> >> doing in >> >> https://lore.kernel.org/git/RFC-patch-v2-4.7-61f4f3d7ef4-20220323T140753Z-avarab@gmail.com/ >> >> is to do likewise for all the PACK-related stuff before we rename it >> >> in-place. Or even have "git gc" issue only a single fsync() for all of >> >> PACKs, their metadata files, commit-graph etc., and then rename() things >> >> in-place as appropriate afterwards. >> >> >> >> diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt >> >> index 365a12dc7ae..536238e209b 100644 >> >> --- a/Documentation/config/core.txt >> >> +++ b/Documentation/config/core.txt >> >> @@ -548,49 +548,35 @@ core.whitespace:: >> >> errors. The default tab width is 8. Allowed values are 1 to 63. >> >> >> >> core.fsync:: >> >> - A comma-separated list of components of the repository that >> >> - should be hardened via the core.fsyncMethod when created or >> >> - modified. You can disable hardening of any component by >> >> - prefixing it with a '-'. Items that are not hardened may be >> >> - lost in the event of an unclean system shutdown. Unless you >> >> - have special requirements, it is recommended that you leave >> >> - this option empty or pick one of `committed`, `added`, >> >> - or `all`. >> >> -+ >> >> -When this configuration is encountered, the set of components starts with >> >> -the platform default value, disabled components are removed, and additional >> >> -components are added. `none` resets the state so that the platform default >> >> -is ignored. >> >> -+ >> >> -The empty string resets the fsync configuration to the platform >> >> -default. The default on most platforms is equivalent to >> >> -`core.fsync=committed,-loose-object`, which has good performance, >> >> -but risks losing recent work in the event of an unclean system shutdown. >> >> -+ >> >> -* `none` clears the set of fsynced components. >> >> -* `loose-object` hardens objects added to the repo in loose-object form. >> >> -* `pack` hardens objects added to the repo in packfile form. >> >> -* `pack-metadata` hardens packfile bitmaps and indexes. >> >> -* `commit-graph` hardens the commit graph file. >> >> -* `index` hardens the index when it is modified. >> >> -* `objects` is an aggregate option that is equivalent to >> >> - `loose-object,pack`. >> >> -* `derived-metadata` is an aggregate option that is equivalent to >> >> - `pack-metadata,commit-graph`. >> >> -* `committed` is an aggregate option that is currently equivalent to >> >> - `objects`. This mode sacrifices some performance to ensure that work >> >> - that is committed to the repository with `git commit` or similar commands >> >> - is hardened. >> >> -* `added` is an aggregate option that is currently equivalent to >> >> - `committed,index`. This mode sacrifices additional performance to >> >> - ensure that the results of commands like `git add` and similar operations >> >> - are hardened. >> >> -* `all` is an aggregate option that syncs all individual components above. >> >> + A boolen defaulting to `true`. To ensure data integrity git >> >> + will fsync() its objects, index and refu updates etc. This can >> >> + be set to `false` to disable `fsync()`-ing. >> >> ++ >> >> +Only set this to `false` if you know what you're doing, and are >> >> +prepared to deal with data corruption. Valid use-cases include >> >> +throwaway uses of repositories on ramdisks, one-off mass-imports >> >> +followed by calling `sync(1)` etc. >> >> ++ >> >> +Note that the syncing of loose objects is currently excluded from >> >> +`core.fsync=true`. To turn on all fsync-ing you'll need >> >> +`core.fsync=true` and `core.fsyncObjectFiles=true`, but see >> >> +`core.fsyncMethod=batch` below for a much faster alternative that's >> >> +just as safe on various modern OS's. >> >> ++ >> >> +The default is in flux and may change in the future, in particular the >> >> +equivalent of the already-deprecated `core.fsyncObjectFiles` setting >> >> +might soon default to `true`, and `core.fsyncMethod`'s default of >> >> +`fsync` might default to a setting deemed to be safe on the local OS, >> >> +suc has `batch` or `writeout-only` >> >> >> >> core.fsyncMethod:: >> >> A value indicating the strategy Git will use to harden repository data >> >> using fsync and related primitives. >> >> + >> >> +Defaults to `fsync`, but as discussed for `core.fsync` above might >> >> +change to use one of the values below taking advantage of >> >> +platform-specific "faster `fsync()`". >> >> ++ >> >> * `fsync` uses the fsync() system call or platform equivalents. >> >> * `writeout-only` issues pagecache writeback requests, but depending on the >> >> filesystem and storage hardware, data added to the repository may not be >> >> @@ -680,8 +666,8 @@ backed up by any standard (e.g. POSIX), but worked in practice on some >> >> Linux setups. >> >> + >> >> Nowadays you should almost certainly want to use >> >> -`core.fsync=loose-object` instead in combination with >> >> -`core.fsyncMethod=bulk`, and possibly with >> >> +`core.fsync=true` instead in combination with >> >> +`core.fsyncMethod=batch`, and possibly with >> >> `fsyncMethod.batch.quarantine=true`, see above. On modern OS's (Linux, >> >> OSX, Windows) that gives you most of the performance benefit of >> >> `core.fsyncObjectFiles=false` with all of the safety of the old >> > >> > I'm at the point where I don't want to endlessly revisit this discussion. >> >> Sorry, my intention isn't to frustrate you, but I do think it's >> important to get this right. >> >> Particularly since this is now in "next", and we're getting closer to a >> release. We can either talk about this now and decide on something, or >> it'll be in a release, and then publicly documented promises will be >> harder to back out of. >> >> I think your suggestion of just hiding the relevant documentation would >> be a good band-aid solution to that. >> >> But I also think that given how I was altering this in my RFC series >> that the premise of how this could be structured has been called into >> question in a way that we didn't (or I don't recall) us having discussed >> before. >> >> I.e. that we can say "sync loose, but not index", or "sync index, but >> not loose" with this config schema. When with "bulk" we it really isn't >> any more expensive to do both if one is true (even cheaper, actually). >> > > I want to make a comment about the Index here. Syncing the index is > strictly required for the "added" level of consistency, so that we > don't lose stuff that leaves the work tree but is staged. But my > Windows enlistment has an index that's 266MB, which would be painful > to sync even with all the optimizations. Maybe with split-index, this > wouldn't be so bad, but I just wanted to call out that some advanced > users may really care about the configurability. So for that use-case you'd like to fsync the loose objects (if any), but not the index? So the FS will "flush" up to the index, and then queue the index for later syncing to platter? But even in that case don't the settings need to be tied to one another, because in the method=bulk sync=index && sync=!loose case wouldn't we be syncing "loose" in any case? > As Git's various database implementations improve, the fsync stuff > will hopefully be more optimal and self-tuning. But as that happens, > Git could just start ignoring settings that lose meaning without tying > anyones hands. Yeah that would alleviate most of my concerns here, but the docs aren't saying anything like that. Since you added them & they just landed, do you mind doing a small follow-up where we e.g. say that these new settings are "EXPERIMENTAL" or whatever, and subject to drastic change? ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: do we have too much fsync() configuration in 'next'? (was: [PATCH v7] core.fsync: documentation and user-friendly aggregate options) 2022-03-26 15:31 ` Ævar Arnfjörð Bjarmason @ 2022-03-27 5:27 ` Neeraj Singh 2022-03-27 12:43 ` Ævar Arnfjörð Bjarmason 0 siblings, 1 reply; 122+ messages in thread From: Neeraj Singh @ 2022-03-27 5:27 UTC (permalink / raw) To: Ævar Arnfjörð Bjarmason Cc: Neeraj K. Singh via GitGitGadget, Johannes Schindelin, Bagas Sanjaya, Git List, Neeraj Singh, Elijah Newren, Patrick Steinhardt, Randall S. Becker, brian m. carlson On Sat, Mar 26, 2022 at 8:34 AM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote: > > > On Fri, Mar 25 2022, Neeraj Singh wrote: > > > On Fri, Mar 25, 2022 at 5:33 PM Ævar Arnfjörð Bjarmason > > <avarab@gmail.com> wrote: > >> > >> > >> On Fri, Mar 25 2022, Neeraj Singh wrote: > >> > >> > On Wed, Mar 23, 2022 at 7:46 AM Ævar Arnfjörð Bjarmason > >> > <avarab@gmail.com> wrote: > >> >> > >> >> > >> >> On Tue, Mar 15 2022, Neeraj Singh wrote: > >> >> > >> >> I know this is probably 80% my fault by egging you on about initially > >> >> adding the wildmatch() based thing you didn't go for. > >> >> > >> >> But having looked at this with fresh eyes quite deeply I really think > >> >> we're severely over-configuring things here: > >> >> > >> >> > +core.fsync:: > >> >> > + A comma-separated list of components of the repository that > >> >> > + should be hardened via the core.fsyncMethod when created or > >> >> > + modified. You can disable hardening of any component by > >> >> > + prefixing it with a '-'. Items that are not hardened may be > >> >> > + lost in the event of an unclean system shutdown. Unless you > >> >> > + have special requirements, it is recommended that you leave > >> >> > + this option empty or pick one of `committed`, `added`, > >> >> > + or `all`. > >> >> > ++ > >> >> > +When this configuration is encountered, the set of components starts with > >> >> > +the platform default value, disabled components are removed, and additional > >> >> > +components are added. `none` resets the state so that the platform default > >> >> > +is ignored. > >> >> > ++ > >> >> > +The empty string resets the fsync configuration to the platform > >> >> > +default. The default on most platforms is equivalent to > >> >> > +`core.fsync=committed,-loose-object`, which has good performance, > >> >> > +but risks losing recent work in the event of an unclean system shutdown. > >> >> > ++ > >> >> > +* `none` clears the set of fsynced components. > >> >> > +* `loose-object` hardens objects added to the repo in loose-object form. > >> >> > +* `pack` hardens objects added to the repo in packfile form. > >> >> > +* `pack-metadata` hardens packfile bitmaps and indexes. > >> >> > +* `commit-graph` hardens the commit graph file. > >> >> > +* `index` hardens the index when it is modified. > >> >> > +* `objects` is an aggregate option that is equivalent to > >> >> > + `loose-object,pack`. > >> >> > +* `derived-metadata` is an aggregate option that is equivalent to > >> >> > + `pack-metadata,commit-graph`. > >> >> > +* `committed` is an aggregate option that is currently equivalent to > >> >> > + `objects`. This mode sacrifices some performance to ensure that work > >> >> > + that is committed to the repository with `git commit` or similar commands > >> >> > + is hardened. > >> >> > +* `added` is an aggregate option that is currently equivalent to > >> >> > + `committed,index`. This mode sacrifices additional performance to > >> >> > + ensure that the results of commands like `git add` and similar operations > >> >> > + are hardened. > >> >> > +* `all` is an aggregate option that syncs all individual components above. > >> >> > + > >> >> > core.fsyncMethod:: > >> >> > A value indicating the strategy Git will use to harden repository data > >> >> > using fsync and related primitives. > >> >> > >> >> On top of my > >> >> https://lore.kernel.org/git/RFC-patch-v2-7.7-a5951366c6e-20220323T140753Z-avarab@gmail.com/ > >> >> which makes the tmp-objdir part of your not-in-next-just-seen follow-up > >> >> series configurable via "fsyncMethod.batch.quarantine" I really think we > >> >> should just go for something like the belwo patch (note that > >> >> misspelled/mistook "bulk" for "batch" in that linked-t patch, fixed > >> >> below. > >> >> > >> >> I.e. I think we should just do our default fsync() of everything, and > >> >> probably SOON make the fsync-ing of loose objects the default. Those who > >> >> care about performance will have "batch" (or "writeout-only"), which we > >> >> can have OS-specific detections for. > >> >> > >> >> But really, all of the rest of this is unduly boxing us into > >> >> overconfiguration that I think nobody really needs. > >> >> > >> > > >> > We've gone over this a few times already, but just wanted to state it > >> > again. The really detailed settings are really there for Git hosters > >> > like GitLab or GitHub. I'd be happy to remove the per-component > >> > core.fsync values from the documentation and leave just the ones we > >> > point the user to. > >> > >> I'm prettty sure (but Patrick knows more) that GitLab's plan for this is > >> to keep it at whatever the safest setting is, presumably GitHub's as > >> well (but I don't know at all on that front). > >> > >> >> If someone really needs this level of detail they can LD_PRELOAD > >> >> something to have fsync intercept fd's and paths, and act appropriately. > >> >> > >> >> Worse, as the RFC series I sent > >> >> (https://lore.kernel.org/git/RFC-cover-v2-0.7-00000000000-20220323T140753Z-avarab@gmail.com/) > >> >> shows we can and should "batch" up fsync() operations across these > >> >> configuration boundaries, which this level of configuration would seem > >> >> to preclude. > >> >> > >> >> Or, we'd need to explain why "core.fsync=loose-object" won't *actually* > >> >> call fsync() on a single loose object's fd under "batch" as I had to do > >> >> on top of this in > >> >> https://lore.kernel.org/git/RFC-patch-v2-6.7-c20301d7967-20220323T140753Z-avarab@gmail.com/ > >> >> > >> > > >> > 99.9% of users don't care and won't look. The ones who do look deeper > >> > and understand the issues have source code and access to this ML > >> > discussion to understand why this works this way. > >> > >> Exactly, so we can hopefully have a simpler interface. > >> > >> >> The same is going to apply for almost all of the rest of these > >> >> configuration categories. > >> >> > >> >> I.e. a natural follow-up to e.g. batching across objects & index as I'm > >> >> doing in > >> >> https://lore.kernel.org/git/RFC-patch-v2-4.7-61f4f3d7ef4-20220323T140753Z-avarab@gmail.com/ > >> >> is to do likewise for all the PACK-related stuff before we rename it > >> >> in-place. Or even have "git gc" issue only a single fsync() for all of > >> >> PACKs, their metadata files, commit-graph etc., and then rename() things > >> >> in-place as appropriate afterwards. > >> >> > >> >> diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt > >> >> index 365a12dc7ae..536238e209b 100644 > >> >> --- a/Documentation/config/core.txt > >> >> +++ b/Documentation/config/core.txt > >> >> @@ -548,49 +548,35 @@ core.whitespace:: > >> >> errors. The default tab width is 8. Allowed values are 1 to 63. > >> >> > >> >> core.fsync:: > >> >> - A comma-separated list of components of the repository that > >> >> - should be hardened via the core.fsyncMethod when created or > >> >> - modified. You can disable hardening of any component by > >> >> - prefixing it with a '-'. Items that are not hardened may be > >> >> - lost in the event of an unclean system shutdown. Unless you > >> >> - have special requirements, it is recommended that you leave > >> >> - this option empty or pick one of `committed`, `added`, > >> >> - or `all`. > >> >> -+ > >> >> -When this configuration is encountered, the set of components starts with > >> >> -the platform default value, disabled components are removed, and additional > >> >> -components are added. `none` resets the state so that the platform default > >> >> -is ignored. > >> >> -+ > >> >> -The empty string resets the fsync configuration to the platform > >> >> -default. The default on most platforms is equivalent to > >> >> -`core.fsync=committed,-loose-object`, which has good performance, > >> >> -but risks losing recent work in the event of an unclean system shutdown. > >> >> -+ > >> >> -* `none` clears the set of fsynced components. > >> >> -* `loose-object` hardens objects added to the repo in loose-object form. > >> >> -* `pack` hardens objects added to the repo in packfile form. > >> >> -* `pack-metadata` hardens packfile bitmaps and indexes. > >> >> -* `commit-graph` hardens the commit graph file. > >> >> -* `index` hardens the index when it is modified. > >> >> -* `objects` is an aggregate option that is equivalent to > >> >> - `loose-object,pack`. > >> >> -* `derived-metadata` is an aggregate option that is equivalent to > >> >> - `pack-metadata,commit-graph`. > >> >> -* `committed` is an aggregate option that is currently equivalent to > >> >> - `objects`. This mode sacrifices some performance to ensure that work > >> >> - that is committed to the repository with `git commit` or similar commands > >> >> - is hardened. > >> >> -* `added` is an aggregate option that is currently equivalent to > >> >> - `committed,index`. This mode sacrifices additional performance to > >> >> - ensure that the results of commands like `git add` and similar operations > >> >> - are hardened. > >> >> -* `all` is an aggregate option that syncs all individual components above. > >> >> + A boolen defaulting to `true`. To ensure data integrity git > >> >> + will fsync() its objects, index and refu updates etc. This can > >> >> + be set to `false` to disable `fsync()`-ing. > >> >> ++ > >> >> +Only set this to `false` if you know what you're doing, and are > >> >> +prepared to deal with data corruption. Valid use-cases include > >> >> +throwaway uses of repositories on ramdisks, one-off mass-imports > >> >> +followed by calling `sync(1)` etc. > >> >> ++ > >> >> +Note that the syncing of loose objects is currently excluded from > >> >> +`core.fsync=true`. To turn on all fsync-ing you'll need > >> >> +`core.fsync=true` and `core.fsyncObjectFiles=true`, but see > >> >> +`core.fsyncMethod=batch` below for a much faster alternative that's > >> >> +just as safe on various modern OS's. > >> >> ++ > >> >> +The default is in flux and may change in the future, in particular the > >> >> +equivalent of the already-deprecated `core.fsyncObjectFiles` setting > >> >> +might soon default to `true`, and `core.fsyncMethod`'s default of > >> >> +`fsync` might default to a setting deemed to be safe on the local OS, > >> >> +suc has `batch` or `writeout-only` > >> >> > >> >> core.fsyncMethod:: > >> >> A value indicating the strategy Git will use to harden repository data > >> >> using fsync and related primitives. > >> >> + > >> >> +Defaults to `fsync`, but as discussed for `core.fsync` above might > >> >> +change to use one of the values below taking advantage of > >> >> +platform-specific "faster `fsync()`". > >> >> ++ > >> >> * `fsync` uses the fsync() system call or platform equivalents. > >> >> * `writeout-only` issues pagecache writeback requests, but depending on the > >> >> filesystem and storage hardware, data added to the repository may not be > >> >> @@ -680,8 +666,8 @@ backed up by any standard (e.g. POSIX), but worked in practice on some > >> >> Linux setups. > >> >> + > >> >> Nowadays you should almost certainly want to use > >> >> -`core.fsync=loose-object` instead in combination with > >> >> -`core.fsyncMethod=bulk`, and possibly with > >> >> +`core.fsync=true` instead in combination with > >> >> +`core.fsyncMethod=batch`, and possibly with > >> >> `fsyncMethod.batch.quarantine=true`, see above. On modern OS's (Linux, > >> >> OSX, Windows) that gives you most of the performance benefit of > >> >> `core.fsyncObjectFiles=false` with all of the safety of the old > >> > > >> > I'm at the point where I don't want to endlessly revisit this discussion. > >> > >> Sorry, my intention isn't to frustrate you, but I do think it's > >> important to get this right. > >> > >> Particularly since this is now in "next", and we're getting closer to a > >> release. We can either talk about this now and decide on something, or > >> it'll be in a release, and then publicly documented promises will be > >> harder to back out of. > >> > >> I think your suggestion of just hiding the relevant documentation would > >> be a good band-aid solution to that. > >> > >> But I also think that given how I was altering this in my RFC series > >> that the premise of how this could be structured has been called into > >> question in a way that we didn't (or I don't recall) us having discussed > >> before. > >> > >> I.e. that we can say "sync loose, but not index", or "sync index, but > >> not loose" with this config schema. When with "bulk" we it really isn't > >> any more expensive to do both if one is true (even cheaper, actually). > >> > > > > I want to make a comment about the Index here. Syncing the index is > > strictly required for the "added" level of consistency, so that we > > don't lose stuff that leaves the work tree but is staged. But my > > Windows enlistment has an index that's 266MB, which would be painful > > to sync even with all the optimizations. Maybe with split-index, this > > wouldn't be so bad, but I just wanted to call out that some advanced > > users may really care about the configurability. > > So for that use-case you'd like to fsync the loose objects (if any), but > not the index? So the FS will "flush" up to the index, and then queue > the index for later syncing to platter? > > > But even in that case don't the settings need to be tied to one another, > because in the method=bulk sync=index && sync=!loose case wouldn't we be > syncing "loose" in any case? > > > As Git's various database implementations improve, the fsync stuff > > will hopefully be more optimal and self-tuning. But as that happens, > > Git could just start ignoring settings that lose meaning without tying > > anyones hands. > > Yeah that would alleviate most of my concerns here, but the docs aren't > saying anything like that. Since you added them & they just landed, do > you mind doing a small follow-up where we e.g. say that these new > settings are "EXPERIMENTAL" or whatever, and subject to drastic change? The doc is already pretty prescriptive. It has this line at the end of the first paragraph: "Unless you have special requirements, it is recommended that you leave this option empty or pick one of `committed`, `added`, or `all`." Those values are already designed to change as Git changes. ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: do we have too much fsync() configuration in 'next'? (was: [PATCH v7] core.fsync: documentation and user-friendly aggregate options) 2022-03-27 5:27 ` Neeraj Singh @ 2022-03-27 12:43 ` Ævar Arnfjörð Bjarmason 2022-03-28 10:56 ` Patrick Steinhardt 0 siblings, 1 reply; 122+ messages in thread From: Ævar Arnfjörð Bjarmason @ 2022-03-27 12:43 UTC (permalink / raw) To: Neeraj Singh Cc: Neeraj K. Singh via GitGitGadget, Johannes Schindelin, Bagas Sanjaya, Git List, Neeraj Singh, Elijah Newren, Patrick Steinhardt, Randall S. Becker, brian m. carlson On Sat, Mar 26 2022, Neeraj Singh wrote: > On Sat, Mar 26, 2022 at 8:34 AM Ævar Arnfjörð Bjarmason > <avarab@gmail.com> wrote: >> >> >> On Fri, Mar 25 2022, Neeraj Singh wrote: >> >> > On Fri, Mar 25, 2022 at 5:33 PM Ævar Arnfjörð Bjarmason >> > <avarab@gmail.com> wrote: [...] >> > I want to make a comment about the Index here. Syncing the index is >> > strictly required for the "added" level of consistency, so that we >> > don't lose stuff that leaves the work tree but is staged. But my >> > Windows enlistment has an index that's 266MB, which would be painful >> > to sync even with all the optimizations. Maybe with split-index, this >> > wouldn't be so bad, but I just wanted to call out that some advanced >> > users may really care about the configurability. >> >> So for that use-case you'd like to fsync the loose objects (if any), but >> not the index? So the FS will "flush" up to the index, and then queue >> the index for later syncing to platter? >> >> >> But even in that case don't the settings need to be tied to one another, >> because in the method=bulk sync=index && sync=!loose case wouldn't we be >> syncing "loose" in any case? >> >> > As Git's various database implementations improve, the fsync stuff >> > will hopefully be more optimal and self-tuning. But as that happens, >> > Git could just start ignoring settings that lose meaning without tying >> > anyones hands. >> >> Yeah that would alleviate most of my concerns here, but the docs aren't >> saying anything like that. Since you added them & they just landed, do >> you mind doing a small follow-up where we e.g. say that these new >> settings are "EXPERIMENTAL" or whatever, and subject to drastic change? > > The doc is already pretty prescriptive. It has this line at the end > of the first paragraph: > "Unless you > have special requirements, it is recommended that you leave > this option empty or pick one of `committed`, `added`, > or `all`." > > Those values are already designed to change as Git changes. I'm referring to the documentation as it stands not being marked as experimental in the sense that we might decide to re-do this to a large extent, i.e. something like the diff I suggested upthread in https://lore.kernel.org/git/220323.86fsn8ohg8.gmgdl@evledraar.gmail.com/ So yes, I agree that it e.g. clearly states that you can add a new core.git=foobar or whatever down the line, but it clearly doesn't suggest that e.g. core.fsync might have boolean semantics in some later version, or that the rest might simply be ignored, even if that e.g. means that we wouldn't sync loose objects on core.fsync=loose-object, as we'd just warn with a "we don't provide this anymore". Or do you disagree with that? IOW I mean that we'd do something like this, either in docs or code: diff --git a/config.c b/config.c index 3c9b6b589ab..94548566073 100644 --- a/config.c +++ b/config.c @@ -1675,6 +1675,9 @@ static int git_default_core_config(const char *var, const char *value, void *cb) } if (!strcmp(var, "core.fsync")) { + if (!the_repository->settings.feature_experimental) + warning(_("the '%s' configuration option is EXPERIMENTAL. opt-in to use it with feature.experimental=true"), + var); if (!value) return config_error_nonbool(var); fsync_components = parse_fsync_components(var, value); @@ -1682,6 +1685,9 @@ static int git_default_core_config(const char *var, const char *value, void *cb) } if (!strcmp(var, "core.fsyncmethod")) { + if (!the_repository->settings.feature_experimental) + warning(_("the '%s' configuration option is EXPERIMENTAL. opt-in to use it with feature.experimental=true"), + var); if (!value) return config_error_nonbool(var); if (!strcmp(value, "fsync")) diff --git a/repo-settings.c b/repo-settings.c index b4fbd16cdcc..f949b65b91e 100644 --- a/repo-settings.c +++ b/repo-settings.c @@ -31,6 +31,7 @@ void prepare_repo_settings(struct repository *r) /* Booleans config or default, cascades to other settings */ repo_cfg_bool(r, "feature.manyfiles", &manyfiles, 0); repo_cfg_bool(r, "feature.experimental", &experimental, 0); + r->settings.feature_experimental = experimental; /* Defaults modified by feature.* */ if (experimental) { diff --git a/repository.h b/repository.h index e29f361703d..db8f99a8989 100644 --- a/repository.h +++ b/repository.h @@ -28,6 +28,7 @@ enum fetch_negotiation_setting { struct repo_settings { int initialized; + int feature_experimental; int core_commit_graph; int commit_graph_read_changed_paths; int gc_write_commit_graph; ^ permalink raw reply related [flat|nested] 122+ messages in thread
* Re: do we have too much fsync() configuration in 'next'? (was: [PATCH v7] core.fsync: documentation and user-friendly aggregate options) 2022-03-27 12:43 ` Ævar Arnfjörð Bjarmason @ 2022-03-28 10:56 ` Patrick Steinhardt 2022-03-28 11:25 ` Ævar Arnfjörð Bjarmason 0 siblings, 1 reply; 122+ messages in thread From: Patrick Steinhardt @ 2022-03-28 10:56 UTC (permalink / raw) To: Ævar Arnfjörð Bjarmason Cc: Neeraj Singh, Neeraj K. Singh via GitGitGadget, Johannes Schindelin, Bagas Sanjaya, Git List, Neeraj Singh, Elijah Newren, Randall S. Becker, brian m. carlson [-- Attachment #1: Type: text/plain, Size: 6443 bytes --] On Sun, Mar 27, 2022 at 02:43:48PM +0200, Ævar Arnfjörð Bjarmason wrote: > > On Sat, Mar 26 2022, Neeraj Singh wrote: > > > On Sat, Mar 26, 2022 at 8:34 AM Ævar Arnfjörð Bjarmason > > <avarab@gmail.com> wrote: > >> > >> > >> On Fri, Mar 25 2022, Neeraj Singh wrote: > >> > >> > On Fri, Mar 25, 2022 at 5:33 PM Ævar Arnfjörð Bjarmason > >> > <avarab@gmail.com> wrote: > [...] > >> > I want to make a comment about the Index here. Syncing the index is > >> > strictly required for the "added" level of consistency, so that we > >> > don't lose stuff that leaves the work tree but is staged. But my > >> > Windows enlistment has an index that's 266MB, which would be painful > >> > to sync even with all the optimizations. Maybe with split-index, this > >> > wouldn't be so bad, but I just wanted to call out that some advanced > >> > users may really care about the configurability. > >> > >> So for that use-case you'd like to fsync the loose objects (if any), but > >> not the index? So the FS will "flush" up to the index, and then queue > >> the index for later syncing to platter? > >> > >> > >> But even in that case don't the settings need to be tied to one another, > >> because in the method=bulk sync=index && sync=!loose case wouldn't we be > >> syncing "loose" in any case? > >> > >> > As Git's various database implementations improve, the fsync stuff > >> > will hopefully be more optimal and self-tuning. But as that happens, > >> > Git could just start ignoring settings that lose meaning without tying > >> > anyones hands. > >> > >> Yeah that would alleviate most of my concerns here, but the docs aren't > >> saying anything like that. Since you added them & they just landed, do > >> you mind doing a small follow-up where we e.g. say that these new > >> settings are "EXPERIMENTAL" or whatever, and subject to drastic change? > > > > The doc is already pretty prescriptive. It has this line at the end > > of the first paragraph: > > "Unless you > > have special requirements, it is recommended that you leave > > this option empty or pick one of `committed`, `added`, > > or `all`." > > > > Those values are already designed to change as Git changes. > > I'm referring to the documentation as it stands not being marked as > experimental in the sense that we might decide to re-do this to a large > extent, i.e. something like the diff I suggested upthread in > https://lore.kernel.org/git/220323.86fsn8ohg8.gmgdl@evledraar.gmail.com/ > > So yes, I agree that it e.g. clearly states that you can add a new > core.git=foobar or whatever down the line, but it clearly doesn't > suggest that e.g. core.fsync might have boolean semantics in some later > version, or that the rest might simply be ignored, even if that > e.g. means that we wouldn't sync loose objects on > core.fsync=loose-object, as we'd just warn with a "we don't provide this > anymore". > > Or do you disagree with that? IOW I mean that we'd do something like > this, either in docs or code: > > diff --git a/config.c b/config.c > index 3c9b6b589ab..94548566073 100644 > --- a/config.c > +++ b/config.c > @@ -1675,6 +1675,9 @@ static int git_default_core_config(const char *var, const char *value, void *cb) > } > > if (!strcmp(var, "core.fsync")) { > + if (!the_repository->settings.feature_experimental) > + warning(_("the '%s' configuration option is EXPERIMENTAL. opt-in to use it with feature.experimental=true"), > + var); > if (!value) > return config_error_nonbool(var); > fsync_components = parse_fsync_components(var, value); > @@ -1682,6 +1685,9 @@ static int git_default_core_config(const char *var, const char *value, void *cb) > } > > if (!strcmp(var, "core.fsyncmethod")) { > + if (!the_repository->settings.feature_experimental) > + warning(_("the '%s' configuration option is EXPERIMENTAL. opt-in to use it with feature.experimental=true"), > + var); > if (!value) > return config_error_nonbool(var); > if (!strcmp(value, "fsync")) Let's please not tie this to `feature.experimental=true`. Setting that option has unintended sideeffects and will also change defaults which we may not want to have in production. I don't mind adding a warning in the docs though that the specific items which can be configured may be subject to change in the future. At GitLab, we've got a three-step plan: 1. We need to migrate to `core.fsync` in the first place. In order to not migrate and change behaviour at the same point in time we already benefit from the fine-grainedness of this config because we can simply say `core.fsync=loose-objects` and have the same behaviour as before with `core.fsyncLooseObjects=true`. 2. We'll want to enable syncing of packfiles, which I think wasn't previously covered by `core.fsyncLooseobjects`. 3. We'll add `refs` to also sync loose refs to disk. So while the end result will be the same as `committed`, having this level of control helps us to assess the impact in a nicer way by being able to do this step by step with feature flags. On the other hand, many of the other parts we don't really care about. Auxiliary metadata like the commit-graph or pack indices are data that can in the worst case be regenerated by us, so it's not clear to me whether it makes to also enable fsyncing those in production. So altogether, I agree with Neeraj: having the fine-grainedness greatly helps us to roll out changes like this and be able to pick what we deem to be important. Personally I would be fine with explicitly pointing out that there are two groups of this config in our docs though: 1. The "porcelain" group: "committed", "added", "all", "none". These are abstract groups whose behaviour should adapt as we change implementations, and are those that should typically be set by a user, if intended. 2. The "plumbing" or "expert" group: these are fine-grained options which shouldn't typically be used by Git users. They still have merit though in hosting environments, where requirements are typically a lot more specific. We may also provide different guarantees for both groups. The first one should definitely be stable, but we might state that the second group is subject to change in the future. Patrick [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: do we have too much fsync() configuration in 'next'? (was: [PATCH v7] core.fsync: documentation and user-friendly aggregate options) 2022-03-28 10:56 ` Patrick Steinhardt @ 2022-03-28 11:25 ` Ævar Arnfjörð Bjarmason 2022-03-28 19:56 ` Neeraj Singh 0 siblings, 1 reply; 122+ messages in thread From: Ævar Arnfjörð Bjarmason @ 2022-03-28 11:25 UTC (permalink / raw) To: Patrick Steinhardt Cc: Neeraj Singh, Neeraj K. Singh via GitGitGadget, Johannes Schindelin, Bagas Sanjaya, Git List, Neeraj Singh, Elijah Newren, Randall S. Becker, brian m. carlson, 'Eric Wong' On Mon, Mar 28 2022, Patrick Steinhardt wrote: > [[PGP Signed Part:Undecided]] > On Sun, Mar 27, 2022 at 02:43:48PM +0200, Ævar Arnfjörð Bjarmason wrote: >> >> On Sat, Mar 26 2022, Neeraj Singh wrote: >> >> > On Sat, Mar 26, 2022 at 8:34 AM Ævar Arnfjörð Bjarmason >> > <avarab@gmail.com> wrote: >> >> >> >> >> >> On Fri, Mar 25 2022, Neeraj Singh wrote: >> >> >> >> > On Fri, Mar 25, 2022 at 5:33 PM Ævar Arnfjörð Bjarmason >> >> > <avarab@gmail.com> wrote: >> [...] >> >> > I want to make a comment about the Index here. Syncing the index is >> >> > strictly required for the "added" level of consistency, so that we >> >> > don't lose stuff that leaves the work tree but is staged. But my >> >> > Windows enlistment has an index that's 266MB, which would be painful >> >> > to sync even with all the optimizations. Maybe with split-index, this >> >> > wouldn't be so bad, but I just wanted to call out that some advanced >> >> > users may really care about the configurability. >> >> >> >> So for that use-case you'd like to fsync the loose objects (if any), but >> >> not the index? So the FS will "flush" up to the index, and then queue >> >> the index for later syncing to platter? >> >> >> >> >> >> But even in that case don't the settings need to be tied to one another, >> >> because in the method=bulk sync=index && sync=!loose case wouldn't we be >> >> syncing "loose" in any case? >> >> >> >> > As Git's various database implementations improve, the fsync stuff >> >> > will hopefully be more optimal and self-tuning. But as that happens, >> >> > Git could just start ignoring settings that lose meaning without tying >> >> > anyones hands. >> >> >> >> Yeah that would alleviate most of my concerns here, but the docs aren't >> >> saying anything like that. Since you added them & they just landed, do >> >> you mind doing a small follow-up where we e.g. say that these new >> >> settings are "EXPERIMENTAL" or whatever, and subject to drastic change? >> > >> > The doc is already pretty prescriptive. It has this line at the end >> > of the first paragraph: >> > "Unless you >> > have special requirements, it is recommended that you leave >> > this option empty or pick one of `committed`, `added`, >> > or `all`." >> > >> > Those values are already designed to change as Git changes. >> >> I'm referring to the documentation as it stands not being marked as >> experimental in the sense that we might decide to re-do this to a large >> extent, i.e. something like the diff I suggested upthread in >> https://lore.kernel.org/git/220323.86fsn8ohg8.gmgdl@evledraar.gmail.com/ >> >> So yes, I agree that it e.g. clearly states that you can add a new >> core.git=foobar or whatever down the line, but it clearly doesn't >> suggest that e.g. core.fsync might have boolean semantics in some later >> version, or that the rest might simply be ignored, even if that >> e.g. means that we wouldn't sync loose objects on >> core.fsync=loose-object, as we'd just warn with a "we don't provide this >> anymore". >> >> Or do you disagree with that? IOW I mean that we'd do something like >> this, either in docs or code: >> >> diff --git a/config.c b/config.c >> index 3c9b6b589ab..94548566073 100644 >> --- a/config.c >> +++ b/config.c >> @@ -1675,6 +1675,9 @@ static int git_default_core_config(const char *var, const char *value, void *cb) >> } >> >> if (!strcmp(var, "core.fsync")) { >> + if (!the_repository->settings.feature_experimental) >> + warning(_("the '%s' configuration option is EXPERIMENTAL. opt-in to use it with feature.experimental=true"), >> + var); >> if (!value) >> return config_error_nonbool(var); >> fsync_components = parse_fsync_components(var, value); >> @@ -1682,6 +1685,9 @@ static int git_default_core_config(const char *var, const char *value, void *cb) >> } >> >> if (!strcmp(var, "core.fsyncmethod")) { >> + if (!the_repository->settings.feature_experimental) >> + warning(_("the '%s' configuration option is EXPERIMENTAL. opt-in to use it with feature.experimental=true"), >> + var); >> if (!value) >> return config_error_nonbool(var); >> if (!strcmp(value, "fsync")) > > Let's please not tie this to `feature.experimental=true`. Setting that > option has unintended sideeffects and will also change defaults which we > may not want to have in production. I don't mind adding a warning in the > docs though that the specific items which can be configured may be > subject to change in the future. Yes, that was a bad (throwaway) idea. I think probably any sort of warning is over-doing it, but having the same in the docs would be good, as in: git gre EXPERIMENTAL -- Documentation > At GitLab, we've got a three-step plan: > > 1. We need to migrate to `core.fsync` in the first place. In order > to not migrate and change behaviour at the same point in time we > already benefit from the fine-grainedness of this config because > we can simply say `core.fsync=loose-objects` and have the same > behaviour as before with `core.fsyncLooseObjects=true`. *nod*. > 2. We'll want to enable syncing of packfiles, which I think wasn't > previously covered by `core.fsyncLooseobjects`. We've always fsynced packfiles and other things that use the finalize_hashfile() API. I.e. the pack metadata (idx,midx,bitmap etc.), commit-graph etc. Which is one thing I find a bit uncomfortable about the proposed config schema, i.e. it's allowing *granular* unsafe behavior we didn't allow before. I think we *should* allow disabling fsync() entirely via: core.fsync=false Per Eric Wong's [added to CC] proposal here: https://lore.kernel.org/git/20211028002102.19384-1-e@80x24.org/; I think that's useful for e.g. running git in CI, one off scripted mass-imports where you run "sync(1)" after (or not...). But I don't really see the use-case for turning off say "index" or "pack-metadata", but otherwise keeping the default fsync(). > 3. We'll add `refs` to also sync loose refs to disk. Maybe I'm reading this wrong, but AFAICT if you upgrade from pre-v2.36.0 to v2.36.0 you'll have no way to do fsync()-ing as it was done before with your bc22d845c43 (core.fsync: new option to harden references, 2022-03-11). I.e. I first thought you meant to start with: core.fsync=-loose-objects core.fsync=-reference And then remove that "core.fsync=-reference" line to get the behavior that'll be new in v2.36.0, but that won't do that. The new "reference" category doesn't just affect loose refs, but all ref updates. So we don't have any way to move to v2.36.0 and get exactly the fsync() behavior we did before, or have I misread the code? Now, I don't think we need it to be configurable at all. I.e. I think your is a bc22d845c43 good change, but it feels weird to make it and leave the default of core.fsyncLooseObjects=false on the table. I.e. what you're summarizing there is true, but it's also true of the loose objects. To the extent that we've had any reason at all not to sync those by default (which has really mostly been "we didn't re-visit it for a while") it's been performance. And both loose refs & loose objects will suffer from the same degradation in performance from many fsync()'s. IOW I think it's perfectly fine not to add a config knob for it other than core.fsync=false, and VERY SOON turn on core.fsyncLooseobjects=true, especially if we can get most of the performance benefits with the "bulk" mode. But why half-way with bc22d845c43? I mean, *that change* should be narrow, but in terms of where we go next what do you think of the above? > So while the end result will be the same as `committed`, having this > level of control helps us to assess the impact in a nicer way by being > able to do this step by step with feature flags. *Nod*, although leaving aside the new syncing of loose refs the plan you outlined above could be done with the proposal of the simpler: core.fsync=true core.fsyncLooseObjects=false > On the other hand, many of the other parts we don't really care about. > Auxiliary metadata like the commit-graph or pack indices are data that > can in the worst case be regenerated by us, so it's not clear to me > whether it makes to also enable fsyncing those in production. I'm not familiar with all of those in detail, i.e. how we behave specifically in the face of them being corrupt. The commit-graph I am, we *should* recover "gracefully" there, but you might have an incident shortly there after due to "for-each-ref --contains" slowdowns by 1000x or whatever. For e.g. *.idx we'd be hosed until manual recovery. So I think those area all in the same bucket as core.fsync=false, and that we don't need the granularity. > So altogether, I agree with Neeraj: having the fine-grainedness greatly > helps us to roll out changes like this and be able to pick what we deem > to be important. Personally I would be fine with explicitly pointing out > that there are two groups of this config in our docs though: Yes, that's fair. But pending replies to the above I think the main point & proposal of us having too much config stands. I.e. depending on what you want to do with loose object refs we'd just need this: core.fsync=[<bool>] # true by default core.fsyncLooseObjects=[<bool>] # false by default core.fsyncLooseRefs=[<bool>] # true by default? And then the "bulk" config, which would be orthagonal to this. I.e. do we really have a use-case for the rest of the kitchen sink? > 1. The "porcelain" group: "committed", "added", "all", "none". These > are abstract groups whose behaviour should adapt as we change > implementations, and are those that should typically be set by a > user, if intended. > > 2. The "plumbing" or "expert" group: these are fine-grained options > which shouldn't typically be used by Git users. They still have > merit though in hosting environments, where requirements are > typically a lot more specific. > > We may also provide different guarantees for both groups. The first one > should definitely be stable, but we might state that the second group is > subject to change in the future. I hope we can work something out :) Overall: I think you've left one of the the main things I brought up[1] unaddressed, i.e. that the core.fsync config schema in its current form assumes that we can sync A or B, and configure those separately. Which AFAIKT is because Neeraj's initial implementation & the discussion was focused on finishing A or B with a per-"group" "cookie" to flush the files. But as [2] shows it's more performant for us to simply defer the fsync of A until the committing of B. Which is the main reason I think we should be re-visiting this. Sure, if we were just syncing A, B or C having per-[ABC] config options might be a bit overdoing it, but would be relatively simple. But once we start using a more optimized version of the "bulk" mode the config schema will be making promises about individual steps in a transaction that I think we'll want to leave opaque, and only promise that when git returns it will have synced all the relevant assets as efficiently as possible. 1. https://lore.kernel.org/git/220323.86fsn8ohg8.gmgdl@evledraar.gmail.com/ 2. https://lore.kernel.org/git/RFC-patch-v2-4.7-61f4f3d7ef4-20220323T140753Z-avarab@gmail.com/ ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: do we have too much fsync() configuration in 'next'? (was: [PATCH v7] core.fsync: documentation and user-friendly aggregate options) 2022-03-28 11:25 ` Ævar Arnfjörð Bjarmason @ 2022-03-28 19:56 ` Neeraj Singh 2022-03-30 16:59 ` Neeraj Singh 0 siblings, 1 reply; 122+ messages in thread From: Neeraj Singh @ 2022-03-28 19:56 UTC (permalink / raw) To: Ævar Arnfjörð Bjarmason Cc: Patrick Steinhardt, Neeraj K. Singh via GitGitGadget, Johannes Schindelin, Bagas Sanjaya, Git List, Neeraj Singh, Elijah Newren, Randall S. Becker, brian m. carlson, Eric Wong On Mon, Mar 28, 2022 at 5:15 AM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote: > > I hope we can work something out :) > > Overall: I think you've left one of the the main things I brought up[1] > unaddressed, i.e. that the core.fsync config schema in its current form > assumes that we can sync A or B, and configure those separately. > > Which AFAIKT is because Neeraj's initial implementation & the discussion > was focused on finishing A or B with a per-"group" "cookie" to flush the > files. > > But as [2] shows it's more performant for us to simply defer the fsync > of A until the committing of B. > > Which is the main reason I think we should be re-visiting this. Sure, if > we were just syncing A, B or C having per-[ABC] config options might be > a bit overdoing it, but would be relatively simple. > > But once we start using a more optimized version of the "bulk" mode the > config schema will be making promises about individual steps in a > transaction that I think we'll want to leave opaque, and only promise > that when git returns it will have synced all the relevant assets as > efficiently as possible. > > 1. https://lore.kernel.org/git/220323.86fsn8ohg8.gmgdl@evledraar.gmail.com/ > 2. https://lore.kernel.org/git/RFC-patch-v2-4.7-61f4f3d7ef4-20220323T140753Z-avarab@gmail.com/ I think the current documentation is fine (obviously since I wrote it). Let's reproduce the first part again: --- core.fsync:: A comma-separated list of components of the repository that should be hardened via the core.fsyncMethod when created or modified. You can disable hardening of any component by prefixing it with a '-'. Items that are not hardened may be lost in the event of an unclean system shutdown. Unless you have special requirements, it is recommended that you leave this option empty or pick one of `committed`, `added`, or `all`. + When this configuration is encountered, the set of components starts with the platform default value, disabled components are removed, and additional components are added. `none` resets the state so that the platform default is ignored. + The empty string resets the fsync configuration to the platform default. The default on most platforms is equivalent to `core.fsync=committed,-loose-object`, which has good performance, but risks losing recent work in the event of an unclean system shutdown. + --- We're only talking about "hardening" parts of the repository, and we say we'll do it using the "fsyncMethod". If you don't harden something, you could lose it if the system dies. All of these statements are true and don't say so much about the implementation of how the hardening happens. It's perfectly valid to not force any component out of the disk cache, and a straightforward implementation of repo transactions can put the sync point anywhere. We also explicitly point the user at a "porcelain" setting in the first paragraph. So I think you're alone in thinking that anything needs to change here. Thanks, Neeraj ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: do we have too much fsync() configuration in 'next'? (was: [PATCH v7] core.fsync: documentation and user-friendly aggregate options) 2022-03-28 19:56 ` Neeraj Singh @ 2022-03-30 16:59 ` Neeraj Singh 0 siblings, 0 replies; 122+ messages in thread From: Neeraj Singh @ 2022-03-30 16:59 UTC (permalink / raw) To: Ævar Arnfjörð Bjarmason Cc: Patrick Steinhardt, Neeraj K. Singh via GitGitGadget, Johannes Schindelin, Bagas Sanjaya, Git List, Neeraj Singh, Elijah Newren, Randall S. Becker, brian m. carlson, Eric Wong On Mon, Mar 28, 2022 at 12:56 PM Neeraj Singh <nksingh85@gmail.com> wrote: > > On Mon, Mar 28, 2022 at 5:15 AM Ævar Arnfjörð Bjarmason > <avarab@gmail.com> wrote: > > So I think you're alone in thinking that anything needs to change here. > Ævar, I apologize for the negative tone and content of this comment. Thanks, Neeraj ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v6 0/6] A design for future-proofing fsync() configuration 2022-03-10 22:43 ` [PATCH v6 0/6] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget ` (5 preceding siblings ...) 2022-03-10 22:43 ` [PATCH v6 6/6] core.fsync: documentation and user-friendly aggregate options Neeraj Singh via GitGitGadget @ 2022-03-10 23:34 ` Junio C Hamano 2022-03-11 0:03 ` Neeraj Singh 2022-03-13 23:50 ` Junio C Hamano 2022-03-11 9:58 ` [PATCH v2] core.fsync: new option to harden references Patrick Steinhardt 7 siblings, 2 replies; 122+ messages in thread From: Junio C Hamano @ 2022-03-10 23:34 UTC (permalink / raw) To: Neeraj K. Singh via GitGitGadget Cc: git, rsbecker, bagasdotme, newren, avarab, nksingh85, ps, sandals, Johannes Schindelin, Neeraj K. Singh "Neeraj K. Singh via GitGitGadget" <gitgitgadget@gmail.com> writes: > After this change, new persistent data files added to the repo will need to > be added to the fsync_component enum and documented in the > Documentation/config/core.txt text. > > V6 changes: > > * Move only the windows csprng includes into wrapper.c rather than all of > them. This fixes the specific build issue due to broken Windows headers. > [6] > * Split the configuration parsing of core.fsync from the mechanism to focus > the review. > * Incorporate Patrick's patch at [7] into the core.fsync mechanism patch. > * Pick the stricter one of core.fsyncObjectFiles and (fsync_components & > FSYNC_COMPONENT_LOOSE_OBJECTS), to respect the older setting. > * Issue a deprecation warning but keep parsing and honoring > core.fsyncObjectFiles. > * Change configuration parsing of core.fsync to always start with the > platform default. none resets to the empty set. The comma separated list > implies a set without regards to ordering now. This follows Junio's > suggestion in [8]. > * Change the documentation of the core.fsync option to reflect the way the > new parsing code works. Hmph, this seems to make one test fail. t5801-remote-helpers.sh (Wstat: 256 Tests: 31 Failed: 4) Failed tests: 14-16, 31 Non-zero exit status: 1 Files=1, Tests=31, 2 wallclock secs ( 0.04 usr 0.00 sys + 1.40 cusr 1.62 csys = 3.06 CPU) Result: FAIL ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v6 0/6] A design for future-proofing fsync() configuration 2022-03-10 23:34 ` [PATCH v6 0/6] A design for future-proofing fsync() configuration Junio C Hamano @ 2022-03-11 0:03 ` Neeraj Singh 2022-03-11 18:50 ` Neeraj Singh 2022-03-13 23:50 ` Junio C Hamano 1 sibling, 1 reply; 122+ messages in thread From: Neeraj Singh @ 2022-03-11 0:03 UTC (permalink / raw) To: Junio C Hamano Cc: Neeraj K. Singh via GitGitGadget, Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Ævar Arnfjörð Bjarmason, Patrick Steinhardt, brian m. carlson, Johannes Schindelin, Neeraj K. Singh On Thu, Mar 10, 2022 at 3:35 PM Junio C Hamano <gitster@pobox.com> wrote: > > "Neeraj K. Singh via GitGitGadget" <gitgitgadget@gmail.com> writes: > > > After this change, new persistent data files added to the repo will need to > > be added to the fsync_component enum and documented in the > > Documentation/config/core.txt text. > > > > V6 changes: > > > > * Move only the windows csprng includes into wrapper.c rather than all of > > them. This fixes the specific build issue due to broken Windows headers. > > [6] > > * Split the configuration parsing of core.fsync from the mechanism to focus > > the review. > > * Incorporate Patrick's patch at [7] into the core.fsync mechanism patch. > > * Pick the stricter one of core.fsyncObjectFiles and (fsync_components & > > FSYNC_COMPONENT_LOOSE_OBJECTS), to respect the older setting. > > * Issue a deprecation warning but keep parsing and honoring > > core.fsyncObjectFiles. > > * Change configuration parsing of core.fsync to always start with the > > platform default. none resets to the empty set. The comma separated list > > implies a set without regards to ordering now. This follows Junio's > > suggestion in [8]. > > * Change the documentation of the core.fsync option to reflect the way the > > new parsing code works. > > Hmph, this seems to make one test fail. > > t5801-remote-helpers.sh (Wstat: 256 Tests: 31 Failed: 4) > Failed tests: 14-16, 31 > Non-zero exit status: 1 > Files=1, Tests=31, 2 wallclock secs ( 0.04 usr 0.00 sys + 1.40 cusr 1.62 csys = 3.06 CPU) > Result: FAIL Thanks for reporting this. I didn't see a failure in CI, nor when running that specific test in mingw. I also munged my config to include core.fsyncObjectFiles and didn't see a failure. Could you please share some more verbose output of the test, so I can look a bit deeper? In parallel, I'm trying again after merging my changes onto seen. Thanks, Neeraj ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v6 0/6] A design for future-proofing fsync() configuration 2022-03-11 0:03 ` Neeraj Singh @ 2022-03-11 18:50 ` Neeraj Singh 0 siblings, 0 replies; 122+ messages in thread From: Neeraj Singh @ 2022-03-11 18:50 UTC (permalink / raw) To: Junio C Hamano Cc: Neeraj K. Singh via GitGitGadget, Git List, Randall S. Becker, Bagas Sanjaya, Elijah Newren, Ævar Arnfjörð Bjarmason, Patrick Steinhardt, brian m. carlson, Johannes Schindelin, Neeraj K. Singh On Thu, Mar 10, 2022 at 4:03 PM Neeraj Singh <nksingh85@gmail.com> wrote: > > On Thu, Mar 10, 2022 at 3:35 PM Junio C Hamano <gitster@pobox.com> wrote: > > > > "Neeraj K. Singh via GitGitGadget" <gitgitgadget@gmail.com> writes: > > > > > After this change, new persistent data files added to the repo will need to > > > be added to the fsync_component enum and documented in the > > > Documentation/config/core.txt text. > > > > > > V6 changes: > > > > > > * Move only the windows csprng includes into wrapper.c rather than all of > > > them. This fixes the specific build issue due to broken Windows headers. > > > [6] > > > * Split the configuration parsing of core.fsync from the mechanism to focus > > > the review. > > > * Incorporate Patrick's patch at [7] into the core.fsync mechanism patch. > > > * Pick the stricter one of core.fsyncObjectFiles and (fsync_components & > > > FSYNC_COMPONENT_LOOSE_OBJECTS), to respect the older setting. > > > * Issue a deprecation warning but keep parsing and honoring > > > core.fsyncObjectFiles. > > > * Change configuration parsing of core.fsync to always start with the > > > platform default. none resets to the empty set. The comma separated list > > > implies a set without regards to ordering now. This follows Junio's > > > suggestion in [8]. > > > * Change the documentation of the core.fsync option to reflect the way the > > > new parsing code works. > > > > Hmph, this seems to make one test fail. > > > > t5801-remote-helpers.sh (Wstat: 256 Tests: 31 Failed: 4) > > Failed tests: 14-16, 31 > > Non-zero exit status: 1 > > Files=1, Tests=31, 2 wallclock secs ( 0.04 usr 0.00 sys + 1.40 cusr 1.62 csys = 3.06 CPU) > > Result: FAIL > > Thanks for reporting this. I didn't see a failure in CI, nor when > running that specific test in mingw. I also munged my config to > include core.fsyncObjectFiles and didn't see a failure. > > Could you please share some more verbose output of the test, so I can > look a bit deeper? In parallel, I'm trying again after merging my > changes onto seen. > > Thanks, > Neeraj Hi Junio, I've also tested v6-on-seen on Linux and I'm still not seeing the failure. Does the failure still happen on your end? Thanks, Neeraj ^ permalink raw reply [flat|nested] 122+ messages in thread
* Re: [PATCH v6 0/6] A design for future-proofing fsync() configuration 2022-03-10 23:34 ` [PATCH v6 0/6] A design for future-proofing fsync() configuration Junio C Hamano 2022-03-11 0:03 ` Neeraj Singh @ 2022-03-13 23:50 ` Junio C Hamano 1 sibling, 0 replies; 122+ messages in thread From: Junio C Hamano @ 2022-03-13 23:50 UTC (permalink / raw) To: Neeraj K. Singh via GitGitGadget Cc: git, rsbecker, bagasdotme, newren, avarab, nksingh85, ps, sandals, Johannes Schindelin, Neeraj K. Singh Junio C Hamano <gitster@pobox.com> writes: > Hmph, this seems to make one test fail. > > t5801-remote-helpers.sh (Wstat: 256 Tests: 31 Failed: 4) > Failed tests: 14-16, 31 > Non-zero exit status: 1 > Files=1, Tests=31, 2 wallclock secs ( 0.04 usr 0.00 sys + 1.40 cusr 1.62 csys = 3.06 CPU) > Result: FAIL False alarm. This byitself, or merged to 'seen' with other random topics, no longer seem to break these tests. ^ permalink raw reply [flat|nested] 122+ messages in thread
* [PATCH v2] core.fsync: new option to harden references 2022-03-10 22:43 ` [PATCH v6 0/6] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget ` (6 preceding siblings ...) 2022-03-10 23:34 ` [PATCH v6 0/6] A design for future-proofing fsync() configuration Junio C Hamano @ 2022-03-11 9:58 ` Patrick Steinhardt 2022-03-25 6:11 ` SZEDER Gábor 7 siblings, 1 reply; 122+ messages in thread From: Patrick Steinhardt @ 2022-03-11 9:58 UTC (permalink / raw) To: git Cc: rsbecker, bagasdotme, newren, avarab, nksingh85, sandals, Neeraj K. Singh, Junio C Hamano [-- Attachment #1: Type: text/plain, Size: 5963 bytes --] When writing both loose and packed references to disk we first create a lockfile, write the updated values into that lockfile, and on commit we rename the file into place. According to filesystem developers, this behaviour is broken because applications should always sync data to disk before doing the final rename to ensure data consistency [1][2][3]. If applications fail to do this correctly, a hard crash of the machine can easily result in corrupted on-disk data. This kind of corruption can in fact be easily observed with Git when the machine hard-resets shortly after writing references to disk. On machines with ext4, this will likely lead to the "empty files" problem: the file has been renamed, but its data has not been synced to disk. The result is that the reference is corrupt, and in the worst case this can lead to data loss. Implement a new option to harden references so that users and admins can avoid this scenario by syncing locked loose and packed references to disk before we rename them into place. [1]: https://thunk.org/tytso/blog/2009/03/15/dont-fear-the-fsync/ [2]: https://btrfs.wiki.kernel.org/index.php/FAQ (What are the crash guarantees of overwrite-by-rename) [3]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/admin-guide/ext4.rst (see auto_da_alloc) Signed-off-by: Patrick Steinhardt <ps@pks.im> --- Hi, here's my updated patch series which implements syncing of refs. It applies on top of Neeraj's v6 of "A design for future-proofing fsync() configuration". I've simplified these patches a bit: - I don't distinguishing between "loose" and "packed" refs anymore. I agree with Junio that it's probably not worth it, but we can still reintroduce the split at a later point without breaking backwards compatibility if the need comes up. - I've simplified the way loose refs are written to disk so that we now sync them when before we close their files. The previous implementation I had was broken because we tried to sync after closing. Because this really only changes a few lines of code I've also decided to squash together the patches into a single one. Patrick Documentation/config/core.txt | 1 + cache.h | 7 +++++-- config.c | 1 + refs/files-backend.c | 1 + refs/packed-backend.c | 3 ++- 5 files changed, 10 insertions(+), 3 deletions(-) diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt index 37105a7be4..812cca7de7 100644 --- a/Documentation/config/core.txt +++ b/Documentation/config/core.txt @@ -575,6 +575,7 @@ but risks losing recent work in the event of an unclean system shutdown. * `index` hardens the index when it is modified. * `objects` is an aggregate option that is equivalent to `loose-object,pack`. +* `reference` hardens references modified in the repo. * `derived-metadata` is an aggregate option that is equivalent to `pack-metadata,commit-graph`. * `committed` is an aggregate option that is currently equivalent to diff --git a/cache.h b/cache.h index cde0900d05..033e5b0779 100644 --- a/cache.h +++ b/cache.h @@ -1005,6 +1005,7 @@ enum fsync_component { FSYNC_COMPONENT_PACK_METADATA = 1 << 2, FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3, FSYNC_COMPONENT_INDEX = 1 << 4, + FSYNC_COMPONENT_REFERENCE = 1 << 5, }; #define FSYNC_COMPONENTS_OBJECTS (FSYNC_COMPONENT_LOOSE_OBJECT | \ @@ -1017,7 +1018,8 @@ enum fsync_component { FSYNC_COMPONENTS_DERIVED_METADATA | \ ~FSYNC_COMPONENT_LOOSE_OBJECT) -#define FSYNC_COMPONENTS_COMMITTED (FSYNC_COMPONENTS_OBJECTS) +#define FSYNC_COMPONENTS_COMMITTED (FSYNC_COMPONENTS_OBJECTS | \ + FSYNC_COMPONENT_REFERENCE) #define FSYNC_COMPONENTS_ADDED (FSYNC_COMPONENTS_COMMITTED | \ FSYNC_COMPONENT_INDEX) @@ -1026,7 +1028,8 @@ enum fsync_component { FSYNC_COMPONENT_PACK | \ FSYNC_COMPONENT_PACK_METADATA | \ FSYNC_COMPONENT_COMMIT_GRAPH | \ - FSYNC_COMPONENT_INDEX) + FSYNC_COMPONENT_INDEX | \ + FSYNC_COMPONENT_REFERENCE) /* * A bitmask indicating which components of the repo should be fsynced. diff --git a/config.c b/config.c index eb75f65338..3c9b6b589a 100644 --- a/config.c +++ b/config.c @@ -1333,6 +1333,7 @@ static const struct fsync_component_name { { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, { "index", FSYNC_COMPONENT_INDEX }, { "objects", FSYNC_COMPONENTS_OBJECTS }, + { "reference", FSYNC_COMPONENT_REFERENCE }, { "derived-metadata", FSYNC_COMPONENTS_DERIVED_METADATA }, { "committed", FSYNC_COMPONENTS_COMMITTED }, { "added", FSYNC_COMPONENTS_ADDED }, diff --git a/refs/files-backend.c b/refs/files-backend.c index f59589d6cc..6521ee8af5 100644 --- a/refs/files-backend.c +++ b/refs/files-backend.c @@ -1787,6 +1787,7 @@ static int write_ref_to_lockfile(struct ref_lock *lock, fd = get_lock_file_fd(&lock->lk); if (write_in_full(fd, oid_to_hex(oid), the_hash_algo->hexsz) < 0 || write_in_full(fd, &term, 1) < 0 || + fsync_component(FSYNC_COMPONENT_REFERENCE, get_lock_file_fd(&lock->lk)) < 0 || close_ref_gently(lock) < 0) { strbuf_addf(err, "couldn't write '%s'", get_lock_file_path(&lock->lk)); diff --git a/refs/packed-backend.c b/refs/packed-backend.c index 27dd8c3922..9d704ccd3e 100644 --- a/refs/packed-backend.c +++ b/refs/packed-backend.c @@ -1262,7 +1262,8 @@ static int write_with_updates(struct packed_ref_store *refs, goto error; } - if (close_tempfile_gently(refs->tempfile)) { + if (fsync_component(FSYNC_COMPONENT_REFERENCE, get_tempfile_fd(refs->tempfile)) || + close_tempfile_gently(refs->tempfile)) { strbuf_addf(err, "error closing file %s: %s", get_tempfile_path(refs->tempfile), strerror(errno)); -- 2.35.1 [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply related [flat|nested] 122+ messages in thread
* Re: [PATCH v2] core.fsync: new option to harden references 2022-03-11 9:58 ` [PATCH v2] core.fsync: new option to harden references Patrick Steinhardt @ 2022-03-25 6:11 ` SZEDER Gábor 0 siblings, 0 replies; 122+ messages in thread From: SZEDER Gábor @ 2022-03-25 6:11 UTC (permalink / raw) To: Patrick Steinhardt Cc: git, rsbecker, bagasdotme, newren, avarab, nksingh85, sandals, Neeraj K. Singh, Junio C Hamano On Fri, Mar 11, 2022 at 10:58:59AM +0100, Patrick Steinhardt wrote: > When writing both loose and packed references to disk we first create a > lockfile, write the updated values into that lockfile, and on commit we > rename the file into place. According to filesystem developers, this > behaviour is broken because applications should always sync data to disk > before doing the final rename to ensure data consistency [1][2][3]. If > applications fail to do this correctly, a hard crash of the machine can > easily result in corrupted on-disk data. > > This kind of corruption can in fact be easily observed with Git when the > machine hard-resets shortly after writing references to disk. On > machines with ext4, this will likely lead to the "empty files" problem: > the file has been renamed, but its data has not been synced to disk. The > result is that the reference is corrupt, and in the worst case this can > lead to data loss. > > Implement a new option to harden references so that users and admins can > avoid this scenario by syncing locked loose and packed references to > disk before we rename them into place. In 't5541-http-push-smart.sh' there is a test case called 'push 2000 tags over http', which does pretty much what it's title says. This patch makes that test case significantly slower. diff --git a/t/t5541-http-push-smart.sh b/t/t5541-http-push-smart.sh index 8ca50f8b18..d7e94cb791 100755 --- a/t/t5541-http-push-smart.sh +++ b/t/t5541-http-push-smart.sh @@ -415,7 +415,7 @@ test_expect_success CMDLINE_LIMIT 'push 2000 tags over http' ' sort | sed "s|.*|$sha1 refs/tags/really-long-tag-name-&|" \ >.git/packed-refs && - run_with_limited_cmdline git push --mirror + run_with_limited_cmdline /usr/bin/time git push --mirror ' test_expect_success GPG 'push with post-receive to inspect certificate' ' Before this patch (bc22d845c4^) 'time' reports: 3.62user 0.03system 0:03.83elapsed 95%CPU (0avgtext+0avgdata 11904maxresident)k 0inputs+312outputs (0major+4597minor)pagefaults 0swaps With this patch (bc22d845c4): 3.56user 0.04system 0:33.60elapsed 10%CPU (0avgtext+0avgdata 11832maxresident)k 0inputs+320outputs (0major+4578minor)pagefaults 0swaps And the total runtime of the whole test script increases from 8-9s to 37-39s. I wonder whether we should relax the fsync options for this test case. > > [1]: https://thunk.org/tytso/blog/2009/03/15/dont-fear-the-fsync/ > [2]: https://btrfs.wiki.kernel.org/index.php/FAQ (What are the crash guarantees of overwrite-by-rename) > [3]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/admin-guide/ext4.rst (see auto_da_alloc) > > Signed-off-by: Patrick Steinhardt <ps@pks.im> > --- > > Hi, > > here's my updated patch series which implements syncing of refs. It > applies on top of Neeraj's v6 of "A design for future-proofing fsync() > configuration". > > I've simplified these patches a bit: > > - I don't distinguishing between "loose" and "packed" refs anymore. > I agree with Junio that it's probably not worth it, but we can > still reintroduce the split at a later point without breaking > backwards compatibility if the need comes up. > > - I've simplified the way loose refs are written to disk so that we > now sync them when before we close their files. The previous > implementation I had was broken because we tried to sync after > closing. > > Because this really only changes a few lines of code I've also decided > to squash together the patches into a single one. > > Patrick > > Documentation/config/core.txt | 1 + > cache.h | 7 +++++-- > config.c | 1 + > refs/files-backend.c | 1 + > refs/packed-backend.c | 3 ++- > 5 files changed, 10 insertions(+), 3 deletions(-) > > diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt > index 37105a7be4..812cca7de7 100644 > --- a/Documentation/config/core.txt > +++ b/Documentation/config/core.txt > @@ -575,6 +575,7 @@ but risks losing recent work in the event of an unclean system shutdown. > * `index` hardens the index when it is modified. > * `objects` is an aggregate option that is equivalent to > `loose-object,pack`. > +* `reference` hardens references modified in the repo. > * `derived-metadata` is an aggregate option that is equivalent to > `pack-metadata,commit-graph`. > * `committed` is an aggregate option that is currently equivalent to > diff --git a/cache.h b/cache.h > index cde0900d05..033e5b0779 100644 > --- a/cache.h > +++ b/cache.h > @@ -1005,6 +1005,7 @@ enum fsync_component { > FSYNC_COMPONENT_PACK_METADATA = 1 << 2, > FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3, > FSYNC_COMPONENT_INDEX = 1 << 4, > + FSYNC_COMPONENT_REFERENCE = 1 << 5, > }; > > #define FSYNC_COMPONENTS_OBJECTS (FSYNC_COMPONENT_LOOSE_OBJECT | \ > @@ -1017,7 +1018,8 @@ enum fsync_component { > FSYNC_COMPONENTS_DERIVED_METADATA | \ > ~FSYNC_COMPONENT_LOOSE_OBJECT) > > -#define FSYNC_COMPONENTS_COMMITTED (FSYNC_COMPONENTS_OBJECTS) > +#define FSYNC_COMPONENTS_COMMITTED (FSYNC_COMPONENTS_OBJECTS | \ > + FSYNC_COMPONENT_REFERENCE) > > #define FSYNC_COMPONENTS_ADDED (FSYNC_COMPONENTS_COMMITTED | \ > FSYNC_COMPONENT_INDEX) > @@ -1026,7 +1028,8 @@ enum fsync_component { > FSYNC_COMPONENT_PACK | \ > FSYNC_COMPONENT_PACK_METADATA | \ > FSYNC_COMPONENT_COMMIT_GRAPH | \ > - FSYNC_COMPONENT_INDEX) > + FSYNC_COMPONENT_INDEX | \ > + FSYNC_COMPONENT_REFERENCE) > > /* > * A bitmask indicating which components of the repo should be fsynced. > diff --git a/config.c b/config.c > index eb75f65338..3c9b6b589a 100644 > --- a/config.c > +++ b/config.c > @@ -1333,6 +1333,7 @@ static const struct fsync_component_name { > { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, > { "index", FSYNC_COMPONENT_INDEX }, > { "objects", FSYNC_COMPONENTS_OBJECTS }, > + { "reference", FSYNC_COMPONENT_REFERENCE }, > { "derived-metadata", FSYNC_COMPONENTS_DERIVED_METADATA }, > { "committed", FSYNC_COMPONENTS_COMMITTED }, > { "added", FSYNC_COMPONENTS_ADDED }, > diff --git a/refs/files-backend.c b/refs/files-backend.c > index f59589d6cc..6521ee8af5 100644 > --- a/refs/files-backend.c > +++ b/refs/files-backend.c > @@ -1787,6 +1787,7 @@ static int write_ref_to_lockfile(struct ref_lock *lock, > fd = get_lock_file_fd(&lock->lk); > if (write_in_full(fd, oid_to_hex(oid), the_hash_algo->hexsz) < 0 || > write_in_full(fd, &term, 1) < 0 || > + fsync_component(FSYNC_COMPONENT_REFERENCE, get_lock_file_fd(&lock->lk)) < 0 || > close_ref_gently(lock) < 0) { > strbuf_addf(err, > "couldn't write '%s'", get_lock_file_path(&lock->lk)); > diff --git a/refs/packed-backend.c b/refs/packed-backend.c > index 27dd8c3922..9d704ccd3e 100644 > --- a/refs/packed-backend.c > +++ b/refs/packed-backend.c > @@ -1262,7 +1262,8 @@ static int write_with_updates(struct packed_ref_store *refs, > goto error; > } > > - if (close_tempfile_gently(refs->tempfile)) { > + if (fsync_component(FSYNC_COMPONENT_REFERENCE, get_tempfile_fd(refs->tempfile)) || > + close_tempfile_gently(refs->tempfile)) { > strbuf_addf(err, "error closing file %s: %s", > get_tempfile_path(refs->tempfile), > strerror(errno)); > -- > 2.35.1 > ^ permalink raw reply related [flat|nested] 122+ messages in thread
end of thread, other threads:[~2022-03-30 17:00 UTC | newest] Thread overview: 122+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-12-04 3:28 [PATCH 0/2] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget 2021-12-04 3:28 ` [PATCH 1/2] fsync: add writeout-only mode for fsyncing repo data Neeraj Singh via GitGitGadget 2021-12-06 7:54 ` Neeraj Singh 2021-12-04 3:28 ` [PATCH 2/2] core.fsync: introduce granular fsync control Neeraj Singh via GitGitGadget 2021-12-07 2:46 ` [PATCH v2 0/3] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget 2021-12-07 2:46 ` [PATCH v2 1/3] core.fsyncmethod: add writeout-only mode Neeraj Singh via GitGitGadget 2021-12-07 11:44 ` Patrick Steinhardt 2021-12-07 12:14 ` Ævar Arnfjörð Bjarmason 2021-12-07 23:29 ` Neeraj Singh 2021-12-07 12:18 ` Ævar Arnfjörð Bjarmason 2021-12-07 23:58 ` Neeraj Singh 2021-12-07 2:46 ` [PATCH v2 2/3] core.fsync: introduce granular fsync control Neeraj Singh via GitGitGadget 2021-12-07 11:53 ` Patrick Steinhardt 2021-12-07 20:46 ` Neeraj Singh 2021-12-07 12:29 ` Ævar Arnfjörð Bjarmason 2021-12-07 21:44 ` Neeraj Singh 2021-12-08 10:05 ` Ævar Arnfjörð Bjarmason 2021-12-09 0:14 ` Neeraj Singh 2021-12-09 0:44 ` Junio C Hamano 2021-12-09 4:08 ` Ævar Arnfjörð Bjarmason 2021-12-09 6:18 ` Neeraj Singh 2022-01-18 23:50 ` Neeraj Singh 2022-01-19 15:28 ` Ævar Arnfjörð Bjarmason 2022-01-19 14:52 ` Ævar Arnfjörð Bjarmason 2022-01-28 1:28 ` Neeraj Singh 2021-12-07 2:46 ` [PATCH v2 3/3] core.fsync: new option to harden the index Neeraj Singh via GitGitGadget 2021-12-07 11:56 ` [PATCH v2 0/3] A design for future-proofing fsync() configuration Patrick Steinhardt 2021-12-08 0:44 ` Neeraj Singh 2021-12-09 0:57 ` [PATCH v3 0/4] " Neeraj K. Singh via GitGitGadget 2021-12-09 0:57 ` [PATCH v3 1/4] core.fsyncmethod: add writeout-only mode Neeraj Singh via GitGitGadget 2021-12-09 0:57 ` [PATCH v3 2/4] core.fsync: introduce granular fsync control Neeraj Singh via GitGitGadget 2021-12-09 0:57 ` [PATCH v3 3/4] core.fsync: new option to harden the index Neeraj Singh via GitGitGadget 2021-12-09 0:57 ` [PATCH v3 4/4] core.fsync: add a `derived-metadata` aggregate option Neeraj Singh via GitGitGadget 2022-01-08 1:13 ` [PATCH v3 0/4] A design for future-proofing fsync() configuration Neeraj Singh 2022-01-09 0:55 ` rsbecker 2022-01-10 19:00 ` Neeraj Singh 2022-02-01 3:33 ` [PATCH v4 " Neeraj K. Singh via GitGitGadget 2022-02-01 3:33 ` [PATCH v4 1/4] core.fsyncmethod: add writeout-only mode Neeraj Singh via GitGitGadget 2022-02-01 3:33 ` [PATCH v4 2/4] core.fsync: introduce granular fsync control Neeraj Singh via GitGitGadget 2022-02-02 0:51 ` Junio C Hamano 2022-02-02 1:42 ` Junio C Hamano 2022-02-11 21:18 ` Neeraj Singh 2022-02-11 22:19 ` Junio C Hamano 2022-02-11 23:04 ` Neeraj Singh 2022-02-11 23:15 ` Junio C Hamano 2022-02-12 0:39 ` rsbecker 2022-02-14 7:04 ` Patrick Steinhardt 2022-02-14 17:17 ` Junio C Hamano 2022-03-09 13:42 ` Patrick Steinhardt 2022-03-09 18:50 ` Ævar Arnfjörð Bjarmason 2022-03-09 20:03 ` Junio C Hamano 2022-03-10 12:33 ` Patrick Steinhardt 2022-03-10 17:15 ` Junio C Hamano 2022-03-09 20:05 ` Neeraj Singh 2022-02-11 20:38 ` Neeraj Singh 2022-02-01 3:33 ` [PATCH v4 3/4] core.fsync: new option to harden the index Neeraj Singh via GitGitGadget 2022-02-01 3:33 ` [PATCH v4 4/4] core.fsync: add a `derived-metadata` aggregate option Neeraj Singh via GitGitGadget 2022-03-09 23:03 ` [PATCH v5 0/5] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget 2022-03-09 23:03 ` [PATCH v5 1/5] wrapper: move inclusion of CSPRNG headers the wrapper.c file Neeraj Singh via GitGitGadget 2022-03-09 23:29 ` Junio C Hamano 2022-03-10 1:21 ` Neeraj Singh 2022-03-10 1:26 ` brian m. carlson 2022-03-10 1:56 ` Neeraj Singh 2022-03-09 23:03 ` [PATCH v5 2/5] core.fsyncmethod: add writeout-only mode Neeraj Singh via GitGitGadget 2022-03-09 23:48 ` Junio C Hamano 2022-03-09 23:03 ` [PATCH v5 3/5] core.fsync: introduce granular fsync control Neeraj Singh via GitGitGadget 2022-03-10 0:21 ` Junio C Hamano 2022-03-10 2:53 ` Neeraj Singh 2022-03-10 7:19 ` Junio C Hamano 2022-03-10 18:38 ` Neeraj Singh 2022-03-10 18:44 ` Junio C Hamano 2022-03-10 19:57 ` Junio C Hamano 2022-03-10 20:25 ` Neeraj Singh 2022-03-10 21:17 ` Junio C Hamano 2022-03-10 13:11 ` Johannes Schindelin 2022-03-10 17:18 ` Junio C Hamano 2022-03-09 23:03 ` [PATCH v5 4/5] core.fsync: new option to harden the index Neeraj Singh via GitGitGadget 2022-03-09 23:03 ` [PATCH v5 5/5] core.fsync: documentation and user-friendly aggregate options Neeraj Singh via GitGitGadget 2022-03-10 9:53 ` Future-proofed syncing of refs Patrick Steinhardt 2022-03-10 9:53 ` [PATCH 6/8] core.fsync: add `fsync_component()` wrapper which doesn't die Patrick Steinhardt 2022-03-10 17:34 ` Junio C Hamano 2022-03-10 18:40 ` Neeraj Singh 2022-03-10 9:53 ` [PATCH 7/8] core.fsync: new option to harden loose references Patrick Steinhardt 2022-03-10 18:25 ` Junio C Hamano 2022-03-10 19:03 ` Neeraj Singh 2022-03-10 22:54 ` Neeraj Singh 2022-03-11 6:40 ` Junio C Hamano 2022-03-11 9:15 ` Patrick Steinhardt 2022-03-11 9:36 ` Ævar Arnfjörð Bjarmason 2022-03-10 9:53 ` [PATCH 8/8] core.fsync: new option to harden packed references Patrick Steinhardt 2022-03-10 18:28 ` Junio C Hamano 2022-03-11 9:10 ` Patrick Steinhardt 2022-03-10 22:43 ` [PATCH v6 0/6] A design for future-proofing fsync() configuration Neeraj K. Singh via GitGitGadget 2022-03-10 22:43 ` [PATCH v6 1/6] wrapper: make inclusion of Windows csprng header tightly scoped Neeraj Singh via GitGitGadget 2022-03-10 22:43 ` [PATCH v6 2/6] core.fsyncmethod: add writeout-only mode Neeraj Singh via GitGitGadget 2022-03-10 22:43 ` [PATCH v6 3/6] core.fsync: introduce granular fsync control infrastructure Neeraj Singh via GitGitGadget 2022-03-10 22:43 ` [PATCH v6 4/6] core.fsync: add configuration parsing Neeraj Singh via GitGitGadget 2022-03-28 11:06 ` Jiang Xin 2022-03-28 19:45 ` Neeraj Singh 2022-03-10 22:43 ` [PATCH v6 5/6] core.fsync: new option to harden the index Neeraj Singh via GitGitGadget 2022-03-10 22:43 ` [PATCH v6 6/6] core.fsync: documentation and user-friendly aggregate options Neeraj Singh via GitGitGadget 2022-03-15 19:12 ` [PATCH v7] " Neeraj Singh 2022-03-15 19:32 ` Junio C Hamano 2022-03-15 19:56 ` Neeraj Singh 2022-03-23 14:20 ` do we have too much fsync() configuration in 'next'? (was: [PATCH v7] core.fsync: documentation and user-friendly aggregate options) Ævar Arnfjörð Bjarmason 2022-03-25 21:24 ` Neeraj Singh 2022-03-26 0:24 ` Ævar Arnfjörð Bjarmason 2022-03-26 1:23 ` do we have too much fsync() configuration in 'next'? Junio C Hamano 2022-03-26 1:25 ` do we have too much fsync() configuration in 'next'? (was: [PATCH v7] core.fsync: documentation and user-friendly aggregate options) Neeraj Singh 2022-03-26 15:31 ` Ævar Arnfjörð Bjarmason 2022-03-27 5:27 ` Neeraj Singh 2022-03-27 12:43 ` Ævar Arnfjörð Bjarmason 2022-03-28 10:56 ` Patrick Steinhardt 2022-03-28 11:25 ` Ævar Arnfjörð Bjarmason 2022-03-28 19:56 ` Neeraj Singh 2022-03-30 16:59 ` Neeraj Singh 2022-03-10 23:34 ` [PATCH v6 0/6] A design for future-proofing fsync() configuration Junio C Hamano 2022-03-11 0:03 ` Neeraj Singh 2022-03-11 18:50 ` Neeraj Singh 2022-03-13 23:50 ` Junio C Hamano 2022-03-11 9:58 ` [PATCH v2] core.fsync: new option to harden references Patrick Steinhardt 2022-03-25 6:11 ` SZEDER Gábor
Code repositories for project(s) associated with this public inbox https://80x24.org/mirrors/git.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).