git@vger.kernel.org mailing list mirror (one of many)
 help / color / mirror / code / Atom feed
From: "Rudy Rigot via GitGitGadget" <gitgitgadget@gmail.com>
To: git@vger.kernel.org
Cc: "Jeff Hostetler" <git@jeffhostetler.com>,
	"Taylor Blau" <me@ttaylorr.com>,
	"Ævar Arnfjörð Bjarmason" <avarab@gmail.com>,
	"Derrick Stolee" <derrickstolee@github.com>,
	"Eric Sunshine" <sunshine@sunshineco.com>,
	"Rudy Rigot" <rudy.rigot@gmail.com>,
	"Rudy Rigot" <rudy.rigot@gmail.com>
Subject: [PATCH v7] status: modernize git-status "slow untracked files" advice
Date: Tue, 22 Nov 2022 16:59:38 +0000	[thread overview]
Message-ID: <pull.1384.v7.git.1669136378754.gitgitgadget@gmail.com> (raw)
In-Reply-To: <pull.1384.v6.git.1668547188070.gitgitgadget@gmail.com>

From: Rudy Rigot <rudy.rigot@gmail.com>

`git status` can be slow when there are a large number of
untracked files and directories since Git must search the entire
worktree to enumerate them.  When it is too slow, Git prints
advice with the elapsed search time and a suggestion to disable
the search using the `-uno` option.  This suggestion also carries
a warning that might scare off some users.

However, these days, `-uno` isn't the only option.  Git can reduce
the size and time of the untracked file search when the
`core.untrackedCache` and `core.fsmonitor` features are enabled by
caching results from previous `git status` invocations.

Therefore, update the `git status` man page to explain the various
configuration options, and update the advice to provide more
detail about the current configuration and to refer to the updated
documentation.

Signed-off-by: Rudy Rigot <rudy.rigot@gmail.com>
---
    status: modernize git-status "slow untracked files" advice
    
    Here is version 7 for this patch.
    
    Changes since v6:
    
     * Readability improvements in tests
     * Rewrite of the commit message

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1384%2Frudyrigot%2Fadvice_statusFsmonitor-v7
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1384/rudyrigot/advice_statusFsmonitor-v7
Pull-Request: https://github.com/gitgitgadget/git/pull/1384

Range-diff vs v6:

 1:  ff3aa0e01c0 ! 1:  871a9becbdf status: long status advice adapted to recent capabilities
     @@ Metadata
      Author: Rudy Rigot <rudy.rigot@gmail.com>
      
       ## Commit message ##
     -    status: long status advice adapted to recent capabilities
     +    status: modernize git-status "slow untracked files" advice
      
     -    Improve the advice displayed when `git status` is slow because
     -    of excessive numbers of untracked files.  Update the `git status`
     -    man page to explain the various configuration options.
     +    `git status` can be slow when there are a large number of
     +    untracked files and directories since Git must search the entire
     +    worktree to enumerate them.  When it is too slow, Git prints
     +    advice with the elapsed search time and a suggestion to disable
     +    the search using the `-uno` option.  This suggestion also carries
     +    a warning that might scare off some users.
      
     -    `git status` can be slow when there are a large number of untracked
     -    files and directories, because Git must search the entire worktree
     -    to enumerate them.  Previously, Git would print an advice message
     -    with the elapsed search time and a suggestion to disable the search
     -    using the `-uno` option.  This suggestion also carried a warning
     -    that might scare off some users.
     +    However, these days, `-uno` isn't the only option.  Git can reduce
     +    the size and time of the untracked file search when the
     +    `core.untrackedCache` and `core.fsmonitor` features are enabled by
     +    caching results from previous `git status` invocations.
      
     -    Git can reduce the size and time of the untracked file search when
     -    the `core.untrackedCache` and `core.fsmonitor` features are enabled
     -    by caching results from previous `git status` invocations.
     -
     -    Update the advice to explain the various combinations of additional
     -    configuration options and refer to (new) documentation in the man
     -    page that explains it in more detail than what can be printed in an
     -    advice message.
     -
     -    Finally, add new tests to verify the new functionality.
     +    Therefore, update the `git status` man page to explain the various
     +    configuration options, and update the advice to provide more
     +    detail about the current configuration and to refer to the updated
     +    documentation.
      
          Signed-off-by: Rudy Rigot <rudy.rigot@gmail.com>
      
     @@ t/t7065-wtstatus-slow.sh (new)
      +
      +test_description='test status when slow untracked files'
      +
     -+. ./test-lib.sh
     ++GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
     ++export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
      +
      +GIT_TEST_UF_DELAY_WARNING=1
      +export GIT_TEST_UF_DELAY_WARNING
      +
     ++. ./test-lib.sh
     ++
      +test_expect_success setup '
     -+	git checkout -b test &&
      +	cat >.gitignore <<-\EOF &&
      +	/actual
      +	/expected
     @@ t/t7065-wtstatus-slow.sh (new)
      +'
      +
      +test_expect_success 'when core.untrackedCache and fsmonitor are unset' '
     -+	test_might_fail git config --unset-all core.untrackedCache &&
     -+	test_might_fail git config --unset-all core.fsmonitor &&
     ++	test_unconfig core.untrackedCache &&
     ++	test_unconfig core.fsmonitor &&
      +	git status >out &&
      +	sed "s/[0-9]\.[0-9][0-9]/X/g" out >actual &&
      +	cat >expected <<-\EOF &&
     -+	On branch test
     ++	On branch main
      +
      +	It took X seconds to enumerate untracked files.
      +	See '"'"'git help status'"'"' for information on how to improve this.
     @@ t/t7065-wtstatus-slow.sh (new)
      +
      +test_expect_success 'when core.untrackedCache true, but not fsmonitor' '
      +	test_config core.untrackedCache true &&
     -+	test_might_fail git config --unset-all core.fsmonitor &&
     ++	test_unconfig core.fsmonitor &&
      +	git status >out &&
      +	sed "s/[0-9]\.[0-9][0-9]/X/g" out >actual &&
      +	cat >expected <<-\EOF &&
     -+	On branch test
     ++	On branch main
      +
      +	It took X seconds to enumerate untracked files.
      +	See '"'"'git help status'"'"' for information on how to improve this.
     @@ t/t7065-wtstatus-slow.sh (new)
      +	git status >out &&
      +	sed "s/[0-9]\.[0-9][0-9]/X/g" out >actual &&
      +	cat >expected <<-\EOF &&
     -+	On branch test
     ++	On branch main
      +
      +	It took X seconds to enumerate untracked files,
      +	but the results were cached, and subsequent runs may be faster.


 Documentation/git-status.txt | 59 +++++++++++++++++++++++++++++
 t/t7065-wtstatus-slow.sh     | 72 ++++++++++++++++++++++++++++++++++++
 wt-status.c                  | 28 +++++++++++---
 3 files changed, 154 insertions(+), 5 deletions(-)
 create mode 100755 t/t7065-wtstatus-slow.sh

diff --git a/Documentation/git-status.txt b/Documentation/git-status.txt
index 5e438a7fdc1..570c36e07c1 100644
--- a/Documentation/git-status.txt
+++ b/Documentation/git-status.txt
@@ -457,6 +457,65 @@ during the write may conflict with other simultaneous processes, causing
 them to fail. Scripts running `status` in the background should consider
 using `git --no-optional-locks status` (see linkgit:git[1] for details).
 
+UNTRACKED FILES AND STATUS SPEED
+--------------------------------
+
+`git status` can be very slow in large worktrees if/when it
+needs to search for untracked files and directories. There are
+many configuration options available to speed this up by either
+avoiding the work or making use of cached results from previous
+Git commands. There is no single optimum set of settings right
+for everyone.  Here is a brief summary of the relevant options
+to help you choose which is right for you.
+
+* First, you may want to run `git status` again. Your current
+	configuration may already be caching `git status` results,
+	so it could be faster on subsequent runs.
+
+* The `--untracked-files=no` flag or the
+	`status.showUntrackedfiles=false` config (see above for both) :
+	indicate that `git status` should not report untracked
+	files. This is the fastest option. `git status` will not list
+	the untracked files, so you need to be careful to remember if
+	you create any new files and manually `git add` them.
+
+* `advice.statusUoption=false` (see linkgit:git-config[1]) :
+	this config option disables a warning message when the search
+	for untracked files takes longer than desired. In some large
+	repositories, this message may appear frequently and not be a
+	helpful signal.
+
+* `core.untrackedCache=true` (see linkgit:git-update-index[1]) :
+	enable the untracked cache feature and only search directories
+	that have been modified since the previous `git status` command.
+	Git remembers the set of untracked files within each directory
+	and assumes that if a directory has not been modified, then
+	the set of untracked files within has not changed.  This is much
+	faster than enumerating the contents of every directory, but still
+	not without cost, because Git still has to search for the set of
+	modified directories. The untracked cache is stored in the
+	`.git/index` file. The reduced cost of searching for untracked
+	files is offset slightly by the increased size of the index and
+	the cost of keeping it up-to-date. That reduced search time is
+	usually worth the additional size.
+
+* `core.untrackedCache=true` and `core.fsmonitor=true` or
+	`core.fsmonitor=<hook_command_pathname>` (see
+	linkgit:git-update-index[1]) : enable both the untracked cache
+	and FSMonitor features and only search directories that have
+	been modified since the previous `git status` command.  This
+	is faster than using just the untracked cache alone because
+	Git can also avoid searching for modified directories.  Git
+	only has to enumerate the exact set of directories that have
+	changed recently. While the FSMonitor feature can be enabled
+	without the untracked cache, the benefits are greatly reduced
+	in that case.
+
+Note that after you turn on the untracked cache and/or FSMonitor
+features it may take a few `git status` commands for the various
+caches to warm up before you see improved command times.  This is
+normal.
+
 SEE ALSO
 --------
 linkgit:gitignore[5]
diff --git a/t/t7065-wtstatus-slow.sh b/t/t7065-wtstatus-slow.sh
new file mode 100755
index 00000000000..6733e8ba36b
--- /dev/null
+++ b/t/t7065-wtstatus-slow.sh
@@ -0,0 +1,72 @@
+#!/bin/sh
+
+test_description='test status when slow untracked files'
+
+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
+
+GIT_TEST_UF_DELAY_WARNING=1
+export GIT_TEST_UF_DELAY_WARNING
+
+. ./test-lib.sh
+
+test_expect_success setup '
+	cat >.gitignore <<-\EOF &&
+	/actual
+	/expected
+	/out
+	EOF
+	git add .gitignore &&
+	git commit -m "Add .gitignore"
+'
+
+test_expect_success 'when core.untrackedCache and fsmonitor are unset' '
+	test_unconfig core.untrackedCache &&
+	test_unconfig core.fsmonitor &&
+	git status >out &&
+	sed "s/[0-9]\.[0-9][0-9]/X/g" out >actual &&
+	cat >expected <<-\EOF &&
+	On branch main
+
+	It took X seconds to enumerate untracked files.
+	See '"'"'git help status'"'"' for information on how to improve this.
+
+	nothing to commit, working tree clean
+	EOF
+	test_cmp expected actual
+'
+
+test_expect_success 'when core.untrackedCache true, but not fsmonitor' '
+	test_config core.untrackedCache true &&
+	test_unconfig core.fsmonitor &&
+	git status >out &&
+	sed "s/[0-9]\.[0-9][0-9]/X/g" out >actual &&
+	cat >expected <<-\EOF &&
+	On branch main
+
+	It took X seconds to enumerate untracked files.
+	See '"'"'git help status'"'"' for information on how to improve this.
+
+	nothing to commit, working tree clean
+	EOF
+	test_cmp expected actual
+'
+
+test_expect_success 'when core.untrackedCache true, and fsmonitor' '
+	test_config core.untrackedCache true &&
+	test_config core.fsmonitor true &&
+	git status >out &&
+	sed "s/[0-9]\.[0-9][0-9]/X/g" out >actual &&
+	cat >expected <<-\EOF &&
+	On branch main
+
+	It took X seconds to enumerate untracked files,
+	but the results were cached, and subsequent runs may be faster.
+	See '"'"'git help status'"'"' for information on how to improve this.
+
+	nothing to commit, working tree clean
+	EOF
+	test_cmp expected actual
+'
+
+test_done
diff --git a/wt-status.c b/wt-status.c
index 5813174896c..1f6d64e759f 100644
--- a/wt-status.c
+++ b/wt-status.c
@@ -18,8 +18,10 @@
 #include "worktree.h"
 #include "lockfile.h"
 #include "sequencer.h"
+#include "fsmonitor-settings.h"
 
 #define AB_DELAY_WARNING_IN_MS (2 * 1000)
+#define UF_DELAY_WARNING_IN_MS (2 * 1000)
 
 static const char cut_line[] =
 "------------------------ >8 ------------------------\n";
@@ -1205,6 +1207,13 @@ static void wt_longstatus_print_tracking(struct wt_status *s)
 	strbuf_release(&sb);
 }
 
+static int uf_was_slow(uint32_t untracked_in_ms)
+{
+	if (getenv("GIT_TEST_UF_DELAY_WARNING"))
+		untracked_in_ms += UF_DELAY_WARNING_IN_MS + 1;
+	return UF_DELAY_WARNING_IN_MS < untracked_in_ms;
+}
+
 static void show_merge_in_progress(struct wt_status *s,
 				   const char *color)
 {
@@ -1814,6 +1823,7 @@ static void wt_longstatus_print(struct wt_status *s)
 {
 	const char *branch_color = color(WT_STATUS_ONBRANCH, s);
 	const char *branch_status_color = color(WT_STATUS_HEADER, s);
+	enum fsmonitor_mode fsm_mode = fsm_settings__get_mode(s->repo);
 
 	if (s->branch) {
 		const char *on_what = _("On branch ");
@@ -1870,13 +1880,21 @@ static void wt_longstatus_print(struct wt_status *s)
 		wt_longstatus_print_other(s, &s->untracked, _("Untracked files"), "add");
 		if (s->show_ignored_mode)
 			wt_longstatus_print_other(s, &s->ignored, _("Ignored files"), "add -f");
-		if (advice_enabled(ADVICE_STATUS_U_OPTION) && 2000 < s->untracked_in_ms) {
+		if (advice_enabled(ADVICE_STATUS_U_OPTION) && uf_was_slow(s->untracked_in_ms)) {
 			status_printf_ln(s, GIT_COLOR_NORMAL, "%s", "");
+			if (fsm_mode > FSMONITOR_MODE_DISABLED) {
+				status_printf_ln(s, GIT_COLOR_NORMAL,
+						_("It took %.2f seconds to enumerate untracked files,\n"
+						"but the results were cached, and subsequent runs may be faster."),
+						s->untracked_in_ms / 1000.0);
+			} else {
+				status_printf_ln(s, GIT_COLOR_NORMAL,
+						_("It took %.2f seconds to enumerate untracked files."),
+						s->untracked_in_ms / 1000.0);
+			}
 			status_printf_ln(s, GIT_COLOR_NORMAL,
-					 _("It took %.2f seconds to enumerate untracked files. 'status -uno'\n"
-					   "may speed it up, but you have to be careful not to forget to add\n"
-					   "new files yourself (see 'git help status')."),
-					 s->untracked_in_ms / 1000.0);
+					_("See 'git help status' for information on how to improve this."));
+			status_printf_ln(s, GIT_COLOR_NORMAL, "%s", "");
 		}
 	} else if (s->committable)
 		status_printf_ln(s, GIT_COLOR_NORMAL, _("Untracked files not listed%s"),

base-commit: 319605f8f00e402f3ea758a02c63534ff800a711
-- 
gitgitgadget

  parent reply	other threads:[~2022-11-22 17:00 UTC|newest]

Thread overview: 58+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-10-15 13:04 [PATCH] fsmonitor: long status advice adapted to the fsmonitor use case Rudy Rigot via GitGitGadget
2022-10-15 13:08 ` Rudy Rigot
2022-10-17 15:39 ` Jeff Hostetler
2022-10-17 16:59   ` Rudy Rigot
2022-10-20 12:56     ` Jeff Hostetler
2022-10-20 20:17       ` Rudy Rigot
2022-10-24 14:55         ` Jeff Hostetler
2022-10-29  0:06 ` [PATCH v2] status: long status advice adapted to recent capabilities Rudy Rigot via GitGitGadget
2022-11-02 19:45   ` Jeff Hostetler
2022-11-02 20:34     ` Rudy Rigot
2022-11-02 23:59     ` Taylor Blau
2022-11-03 14:28       ` Rudy Rigot
2022-11-04  8:52         ` Ævar Arnfjörð Bjarmason
2022-11-04 15:33           ` Rudy Rigot
2022-11-04 21:38         ` Taylor Blau
2022-11-02 21:27   ` [PATCH v3] " Rudy Rigot via GitGitGadget
2022-11-04 21:40     ` Taylor Blau
2022-11-07 20:02       ` Derrick Stolee
2022-11-07 23:19         ` Taylor Blau
2022-11-15 16:38       ` Jeff Hostetler
2022-11-07 20:01     ` Derrick Stolee
2022-11-07 20:20       ` Eric Sunshine
2022-11-07 20:31         ` Rudy Rigot
2022-11-10  4:46     ` [PATCH v4] " Rudy Rigot via GitGitGadget
2022-11-10  5:42       ` Eric Sunshine
2022-11-10 17:01         ` Rudy Rigot
2022-11-10 17:30           ` Eric Sunshine
2022-11-10 17:47             ` Rudy Rigot
2022-11-10 20:04       ` [PATCH v5] " Rudy Rigot via GitGitGadget
2022-11-15 16:39         ` Jeff Hostetler
2022-11-15 16:42           ` Rudy Rigot
2022-11-15 17:26         ` Eric Sunshine
2022-11-15 17:45           ` Rudy Rigot
2022-11-15 18:06             ` Eric Sunshine
2022-11-15 18:08               ` Rudy Rigot
2022-11-15 21:19         ` [PATCH v6] " Rudy Rigot via GitGitGadget
2022-11-21  5:06           ` Eric Sunshine
2022-11-21 15:54             ` Rudy Rigot
2022-11-21 16:17               ` Eric Sunshine
2022-11-22 16:52                 ` Rudy Rigot
2022-11-22 17:18                   ` Eric Sunshine
2022-11-22 17:24                     ` Eric Sunshine
2022-11-22 17:29                       ` Rudy Rigot
2022-11-22 17:40                     ` Eric Sunshine
2022-11-22 18:07                       ` Eric Sunshine
2022-11-22 19:19                         ` Rudy Rigot
2022-11-22 19:48                           ` Eric Sunshine
2022-11-22 16:59           ` Rudy Rigot via GitGitGadget [this message]
2022-11-22 22:07             ` [PATCH v8] status: modernize git-status "slow untracked files" advice Rudy Rigot via GitGitGadget
2022-11-25  4:58               ` Junio C Hamano
2022-11-29 15:21                 ` Rudy Rigot
2022-11-30  0:51                   ` Rudy Rigot
2022-11-30  0:52               ` [PATCH v9] " Rudy Rigot via GitGitGadget
2022-12-01  6:48                 ` Junio C Hamano
2022-12-01 15:16                   ` Rudy Rigot
2022-12-01 22:45                     ` Junio C Hamano
2022-12-01 22:57                       ` Rudy Rigot
2023-05-11  5:17               ` [PATCH v8] " Eric Sunshine

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=pull.1384.v7.git.1669136378754.gitgitgadget@gmail.com \
    --to=gitgitgadget@gmail.com \
    --cc=avarab@gmail.com \
    --cc=derrickstolee@github.com \
    --cc=git@jeffhostetler.com \
    --cc=git@vger.kernel.org \
    --cc=me@ttaylorr.com \
    --cc=rudy.rigot@gmail.com \
    --cc=sunshine@sunshineco.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).