[PATCH v3] 0/4 fsmonitor fixes

[PATCH v3] 0/4 fsmonitor fixes

[Alex Vandiver]

Updates since v2:

 - Fix tab which crept into 1/4

 - Fixed the benchmarking code in the commit message in 2/4 to just
   always load JSON::XS -- the previous version was the version where
   I'd broken that to force loading of JSON::PP.

 - Remove the --no-pretty from the t/ version of query-watchman in
   2/4; I don't know how I messed up diff'ing the file previously, but
   if there are already differences, it makes sense to let them slide.

[PATCH v3 1/4] fsmonitor: Set the PWD to the top of the working tree

[Alex Vandiver]

The fsmonitor command inherits the PWD of its caller, which may be
anywhere in the working copy.  This makes is difficult for the
fsmonitor command to operate on the whole repository.  Specifically,
for the watchman integration, this causes each subdirectory to get its
own watch entry.

Set the CWD to the top of the working directory, for consistency.

Signed-off-by: Alex Vandiver <alexmv@dropbox.com>
---
 fsmonitor.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/fsmonitor.c b/fsmonitor.c
index 7c1540c05..4ea44dcc6 100644
--- a/fsmonitor.c
+++ b/fsmonitor.c
@@ -121,6 +121,7 @@ static int query_fsmonitor(int version, uint64_t last_update, struct strbuf *que
 	argv[3] = NULL;
 	cp.argv = argv;
 	cp.use_shell = 1;
+	cp.dir = get_git_work_tree();
 
 	return capture_command(&cp, query_result, 1024);
 }
-- 
2.15.0.rc1.413.g76aedb451

[PATCH v3 2/4] fsmonitor: Don't bother pretty-printing JSON from watchman

[Alex Vandiver]

This provides modest performance savings.  Benchmarking with the
following program, with and without `--no-pretty`, we find savings of
23% (0.316s -> 0.242s) in the git repository, and savings of 8% (5.24s
-> 4.86s) on a large repository with 580k files in the working copy.

    #!/usr/bin/perl

    use strict;
    use warnings;
    use IPC::Open2;
    use JSON::XS;

    my $pid = open2(\*CHLD_OUT, \*CHLD_IN, "watchman -j @ARGV")
        or die "open2() failed: $!\n" .
        "Falling back to scanning...\n";

    my $query = qq|["query", "$ENV{PWD}", {}]|;

    print CHLD_IN $query;
    close CHLD_IN;
    my $response = do {local $/; <CHLD_OUT>};

    JSON::XS->new->utf8->decode($response);

Signed-off-by: Alex Vandiver <alexmv@dropbox.com>
---
 templates/hooks--fsmonitor-watchman.sample | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/templates/hooks--fsmonitor-watchman.sample b/templates/hooks--fsmonitor-watchman.sample
index 9eba8a740..9a082f278 100755
--- a/templates/hooks--fsmonitor-watchman.sample
+++ b/templates/hooks--fsmonitor-watchman.sample
@@ -49,7 +49,7 @@ launch_watchman();
 
 sub launch_watchman {
 
-	my $pid = open2(\*CHLD_OUT, \*CHLD_IN, 'watchman -j')
+	my $pid = open2(\*CHLD_OUT, \*CHLD_IN, 'watchman -j --no-pretty')
 	    or die "open2() failed: $!\n" .
 	    "Falling back to scanning...\n";
 
-- 
2.15.0.rc1.413.g76aedb451

[PATCH v3 3/4] fsmonitor: Document GIT_TRACE_FSMONITOR

[Alex Vandiver]

Signed-off-by: Alex Vandiver <alexmv@dropbox.com>
---
 Documentation/git.txt | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/Documentation/git.txt b/Documentation/git.txt
index 1fca63634..720db196e 100644
--- a/Documentation/git.txt
+++ b/Documentation/git.txt
@@ -594,6 +594,10 @@ into it.
 Unsetting the variable, or setting it to empty, "0" or
 "false" (case insensitive) disables trace messages.
 
+`GIT_TRACE_FSMONITOR`::
+	Enables trace messages for the filesystem monitor extension.
+	See `GIT_TRACE` for available trace output options.
+
 `GIT_TRACE_PACK_ACCESS`::
 	Enables trace messages for all accesses to any packs. For each
 	access, the pack file name and an offset in the pack is
-- 
2.15.0.rc1.413.g76aedb451

[PATCH v3 4/4] fsmonitor: Delay updating state until after split index is merged

[Alex Vandiver]

If the fsmonitor extension is used in conjunction with the split index
extension, the set of entries in the index when it is first loaded is
only a subset of the real index.  This leads to only the non-"base"
index being marked as CE_FSMONITOR_VALID.

Delay the expansion of the ewah bitmap until after tweak_split_index
has been called to merge in the base index as well.

The new fsmonitor_dirty is kept from being leaked by dint of being
cleaned up in post_read_index_from, which is guaranteed to be called
after do_read_index in read_index_from.

Signed-off-by: Alex Vandiver <alexmv@dropbox.com>
---
 cache.h     |  1 +
 fsmonitor.c | 39 ++++++++++++++++++++++++---------------
 2 files changed, 25 insertions(+), 15 deletions(-)

diff --git a/cache.h b/cache.h
index 25adcf681..0a4f43ec2 100644
--- a/cache.h
+++ b/cache.h
@@ -348,6 +348,7 @@ struct index_state {
 	unsigned char sha1[20];
 	struct untracked_cache *untracked;
 	uint64_t fsmonitor_last_update;
+	struct ewah_bitmap *fsmonitor_dirty;
 };
 
 extern struct index_state the_index;
diff --git a/fsmonitor.c b/fsmonitor.c
index 4ea44dcc6..417759224 100644
--- a/fsmonitor.c
+++ b/fsmonitor.c
@@ -49,20 +49,7 @@ int read_fsmonitor_extension(struct index_state *istate, const void *data,
 		ewah_free(fsmonitor_dirty);
 		return error("failed to parse ewah bitmap reading fsmonitor index extension");
 	}
-
-	if (git_config_get_fsmonitor()) {
-		/* Mark all entries valid */
-		for (i = 0; i < istate->cache_nr; i++)
-			istate->cache[i]->ce_flags |= CE_FSMONITOR_VALID;
-
-		/* Mark all previously saved entries as dirty */
-		ewah_each_bit(fsmonitor_dirty, fsmonitor_ewah_callback, istate);
-
-		/* Now mark the untracked cache for fsmonitor usage */
-		if (istate->untracked)
-			istate->untracked->use_fsmonitor = 1;
-	}
-	ewah_free(fsmonitor_dirty);
+	istate->fsmonitor_dirty = fsmonitor_dirty;
 
 	trace_printf_key(&trace_fsmonitor, "read fsmonitor extension successful");
 	return 0;
@@ -239,7 +226,29 @@ void remove_fsmonitor(struct index_state *istate)
 
 void tweak_fsmonitor(struct index_state *istate)
 {
-	switch (git_config_get_fsmonitor()) {
+	int i;
+	int fsmonitor_enabled = git_config_get_fsmonitor();
+
+	if (istate->fsmonitor_dirty) {
+		if (fsmonitor_enabled) {
+			/* Mark all entries valid */
+			for (i = 0; i < istate->cache_nr; i++) {
+				istate->cache[i]->ce_flags |= CE_FSMONITOR_VALID;
+			}
+
+			/* Mark all previously saved entries as dirty */
+			ewah_each_bit(istate->fsmonitor_dirty, fsmonitor_ewah_callback, istate);
+
+			/* Now mark the untracked cache for fsmonitor usage */
+			if (istate->untracked)
+				istate->untracked->use_fsmonitor = 1;
+		}
+
+		ewah_free(istate->fsmonitor_dirty);
+		istate->fsmonitor_dirty = NULL;
+	}
+
+	switch (fsmonitor_enabled) {
 	case -1: /* keep: do nothing */
 		break;
 	case 0: /* false */
-- 
2.15.0.rc1.413.g76aedb451

Re: [PATCH v3] 0/4 fsmonitor fixes

[Johannes Schindelin]

Hi Alex,

On Fri, 27 Oct 2017, Alex Vandiver wrote:

> Updates since v2:
> 
>  - Fix tab which crept into 1/4
> 
>  - Fixed the benchmarking code in the commit message in 2/4 to just
>    always load JSON::XS -- the previous version was the version where
>    I'd broken that to force loading of JSON::PP.
> 
>  - Remove the --no-pretty from the t/ version of query-watchman in
>    2/4; I don't know how I messed up diff'ing the file previously, but
>    if there are already differences, it makes sense to let them slide.

Sounds good!
Dscho

Re: [PATCH v3] 0/4 fsmonitor fixes

[Ben Peart]

On 10/27/2017 7:26 PM, Alex Vandiver wrote:
> Updates since v2:
> 
>   - Fix tab which crept into 1/4
> 
>   - Fixed the benchmarking code in the commit message in 2/4 to just
>     always load JSON::XS -- the previous version was the version where
>     I'd broken that to force loading of JSON::PP.
> 
>   - Remove the --no-pretty from the t/ version of query-watchman in
>     2/4; I don't know how I messed up diff'ing the file previously, but
>     if there are already differences, it makes sense to let them slide.
> 

Thanks Alex, nice improvements.  This looks good to go.

Re: [PATCH v3 4/4] fsmonitor: Delay updating state until after split index is merged

[Junio C Hamano]

Alex Vandiver <alexmv@dropbox.com> writes:

> diff --git a/fsmonitor.c b/fsmonitor.c
> index 4ea44dcc6..417759224 100644
> --- a/fsmonitor.c
> +++ b/fsmonitor.c
> @@ -49,20 +49,7 @@ int read_fsmonitor_extension(struct index_state *istate, const void *data,
>  		ewah_free(fsmonitor_dirty);
>  		return error("failed to parse ewah bitmap reading fsmonitor index extension");
>  	}
> -
> -	if (git_config_get_fsmonitor()) {
> -		/* Mark all entries valid */
> -		for (i = 0; i < istate->cache_nr; i++)
> -			istate->cache[i]->ce_flags |= CE_FSMONITOR_VALID;
> -
> -		/* Mark all previously saved entries as dirty */
> -		ewah_each_bit(fsmonitor_dirty, fsmonitor_ewah_callback, istate);
> -
> -		/* Now mark the untracked cache for fsmonitor usage */
> -		if (istate->untracked)
> -			istate->untracked->use_fsmonitor = 1;
> -	}
> -	ewah_free(fsmonitor_dirty);
> +	istate->fsmonitor_dirty = fsmonitor_dirty;

This makes local variable "int i;" in this function unused and gets
compiler warning.

Re: [PATCH v3 4/4] fsmonitor: Delay updating state until after split index is merged

[Johannes Schindelin]

Hi,

On Tue, 31 Oct 2017, Junio C Hamano wrote:

> Alex Vandiver <alexmv@dropbox.com> writes:
> 
> > diff --git a/fsmonitor.c b/fsmonitor.c
> > index 4ea44dcc6..417759224 100644
> > --- a/fsmonitor.c
> > +++ b/fsmonitor.c
> > @@ -49,20 +49,7 @@ int read_fsmonitor_extension(struct index_state *istate, const void *data,
> >  		ewah_free(fsmonitor_dirty);
> >  		return error("failed to parse ewah bitmap reading fsmonitor index extension");
> >  	}
> > -
> > -	if (git_config_get_fsmonitor()) {
> > -		/* Mark all entries valid */
> > -		for (i = 0; i < istate->cache_nr; i++)
> > -			istate->cache[i]->ce_flags |= CE_FSMONITOR_VALID;
> > -
> > -		/* Mark all previously saved entries as dirty */
> > -		ewah_each_bit(fsmonitor_dirty, fsmonitor_ewah_callback, istate);
> > -
> > -		/* Now mark the untracked cache for fsmonitor usage */
> > -		if (istate->untracked)
> > -			istate->untracked->use_fsmonitor = 1;
> > -	}
> > -	ewah_free(fsmonitor_dirty);
> > +	istate->fsmonitor_dirty = fsmonitor_dirty;
> 
> This makes local variable "int i;" in this function unused and gets
> compiler warning.

... to which end we introduced the DEVELOPER flag to catch these: if you
call

	make DEVELOPER=1

and compile with GCC or Clang, it will elevate such warnings to errors,
and we highly encourage contributors to build their patched source code
with said flag.

Thanks,
Johannes

Re: [PATCH v3 4/4] fsmonitor: Delay updating state until after split index is merged

[Alex Vandiver]

On Tue, 31 Oct 2017, Junio C Hamano wrote:
> This makes local variable "int i;" in this function unused and gets
> compiler warning.

Apologies for leaving that detritus -- I saw you added a 'SQUASH??' commit
to deal with it, which LGTM.

On Tue, 31 Oct 2017, Johannes Schindelin wrote:
> ... to which end we introduced the DEVELOPER flag to catch these: if you
> call
> 
> 	make DEVELOPER=1

Aha!  Thanks for the tip; I'll be sure to use that from now on.
 - Alex