Missing file in 2.23 (p5302-pack-index.subtests)?

Missing file in 2.23 (p5302-pack-index.subtests)?

[Theodore Y. Ts'o]

I was trying to run "make profile" on the master branch (commit
5fa0f5238b: "Git 2.23") and it died in the

	$(MAKE) PROFILE=GEN perf

dies with:

	cannot open test-results/p5302-pack-index.subtests: No such file or directory at ./aggregate.perl line 153.

I presume that's becuase the file t/perf/test-results/p5302-pack-index.subtests is missing?

						- Ted

Re: Missing file in 2.23 (p5302-pack-index.subtests)?

[Jeff King]

On Sun, Aug 18, 2019 at 12:03:17PM -0400, Theodore Y. Ts'o wrote:

> I was trying to run "make profile" on the master branch (commit
> 5fa0f5238b: "Git 2.23") and it died in the
> 
> 	$(MAKE) PROFILE=GEN perf
> 
> dies with:
> 
> 	cannot open test-results/p5302-pack-index.subtests: No such file or directory at ./aggregate.perl line 153.
> 
> I presume that's becuase the file t/perf/test-results/p5302-pack-index.subtests is missing?

That file should be created by running p5302. If you do:

  cd t/perf
  rm -rf test-results
  ./run p5302-pack-index.sh

is it generated?

If so, then perhaps something removed it while the perf tests were
running. Did you run perhaps run "make clean" simultaneously?

-Peff

Re: Missing file in 2.23 (p5302-pack-index.subtests)?

[Jeff King]

On Mon, Aug 26, 2019 at 04:50:13PM -0400, Jeff King wrote:

> On Sun, Aug 18, 2019 at 12:03:17PM -0400, Theodore Y. Ts'o wrote:
> 
> > I was trying to run "make profile" on the master branch (commit
> > 5fa0f5238b: "Git 2.23") and it died in the
> > 
> > 	$(MAKE) PROFILE=GEN perf
> > 
> > dies with:
> > 
> > 	cannot open test-results/p5302-pack-index.subtests: No such file or directory at ./aggregate.perl line 153.
> > 
> > I presume that's becuase the file t/perf/test-results/p5302-pack-index.subtests is missing?
> 
> That file should be created by running p5302. If you do:

By the way, I tried "make perf" as you specified above, and it worked
fine for me. And I don't think there are any changes in v2.23 that
should touch this area. But certainly it would be interesting to know if
it fails consistently for you, and if so, whether older versions work
(so we can bisect).

-Peff

Re: Missing file in 2.23 (p5302-pack-index.subtests)?

[Theodore Y. Ts'o]

On Mon, Aug 26, 2019 at 04:50:13PM -0400, Jeff King wrote:
> On Sun, Aug 18, 2019 at 12:03:17PM -0400, Theodore Y. Ts'o wrote:
> 
> > I was trying to run "make profile" on the master branch (commit
> > 5fa0f5238b: "Git 2.23") and it died in the
> > 
> > 	$(MAKE) PROFILE=GEN perf
> > 
> > dies with:
> > 
> > 	cannot open test-results/p5302-pack-index.subtests: No such file or directory at ./aggregate.perl line 153.
> > 
> > I presume that's becuase the file t/perf/test-results/p5302-pack-index.subtests is missing?
> 
> That file should be created by running p5302. If you do:
> 
>   cd t/perf
>   rm -rf test-results
>   ./run p5302-pack-index.sh
> 
> is it generated?

No, it wasn't.  See below....

					- Ted

<tytso@lambda> {/usr/projects/git/git-core/t/perf}   (master)
1140% git show
commit 5fa0f5238b0cd46cfe7f6fa76c3f526ea98148d9 (HEAD -> master, tag: v2.23.0, origin/master, origin/HEAD)
Author: Junio C Hamano <gitster@pobox.com>
Date:   Fri Aug 16 10:28:23 2019 -0700

    Git 2.23
    
    Signed-off-by: Junio C Hamano <gitster@pobox.com>

diff --git a/GIT-VERSION-GEN b/GIT-VERSION-GEN
index 551e607e73..a1539a7ce6 100755
--- a/GIT-VERSION-GEN
+++ b/GIT-VERSION-GEN
@@ -1,7 +1,7 @@
 #!/bin/sh
 
 GVF=GIT-VERSION-FILE
-DEF_VER=v2.23.0-rc2
+DEF_VER=v2.23.0
 
 LF='
 '
<tytso@lambda> {/usr/projects/git/git-core/t/perf}   (master)
1141% rm -rf test-results ; ./run p5302-pack-index.sh
=== Running 1 tests in this tree ===
warning: $GIT_PERF_LARGE_REPO is $GIT_BUILD_DIR.
warning: This will work, but may not be a sufficiently large repo
warning: for representative measurements.
not ok 1 - repack
#	
#		git repack -ad &&
#		PACK=$(ls .git/objects/pack/*.pack | head -n1) &&
#		test -f "$PACK" &&
#		export PACK
#	
cannot open test-results/p5302-pack-index.subtests: No such file or directory at ./aggregate.perl line 153.
<tytso@lambda> {/usr/projects/git/git-core/t/perf}   (master)
1142% git status
On branch master
Your branch is up to date with 'origin/master'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	../../MAKELOG
	../../MAKELOG.1
	../../common-cmds.h
	../../git-lost-found
	../../git-peek-remote
	../../git-rebase--am
	../../git-rebase--common
	../../git-rebase--helper
	../../git-rebase--interactive
	../../git-rebase--merge
	../../git-relink
	../../git-remote-testgit
	../../git-repo-config
	../../git-tar-tree
	../../git_remote_helpers/
	../../perl/MYMETA.json
	../../perl/MYMETA.yml
	../../perl/PM.stamp
	../../perl/blib/
	../../perl/perl.mak
	../../perl/pm_to_blib
	../../test-chmtime
	../../test-config
	../../test-credential
	../../test-ctype
	../../test-date
	../../test-delta
	../../test-dump-cache-tree
	../../test-dump-split-index
	../../test-dump-untracked-cache
	../../test-fake-ssh
	../../test-genrandom
	../../test-hashmap
	../../test-index-version
	../../test-line-buffer
	../../test-match-trees
	../../test-mergesort
	../../test-mktemp
	../../test-obj-pool
	../../test-parse-options
	../../test-path-utils
	../../test-prio-queue
	../../test-read-cache
	../../test-regex
	../../test-revision-walking
	../../test-run-command
	../../test-scrap-cache-tree
	../../test-sha1
	../../test-sha1-array
	../../test-sigchain
	../../test-string-list
	../../test-string-pool
	../../test-submodule-config
	../../test-subprocess
	../../test-svn-fe
	../../test-treap
	../../test-urlmatch-normalization
	../../test-wildmatch

nothing added to commit but untracked files present (use "git add" to track)

Re: Missing file in 2.23 (p5302-pack-index.subtests)?

[Jeff King]

On Mon, Aug 26, 2019 at 09:29:42PM -0400, Theodore Y. Ts'o wrote:

> > > I presume that's becuase the file t/perf/test-results/p5302-pack-index.subtests is missing?
> > 
> > That file should be created by running p5302. If you do:
> > 
> >   cd t/perf
> >   rm -rf test-results
> >   ./run p5302-pack-index.sh
> > 
> > is it generated?
> 
> No, it wasn't.  See below....

Weird. The output here:

> 1141% rm -rf test-results ; ./run p5302-pack-index.sh
> === Running 1 tests in this tree ===
> warning: $GIT_PERF_LARGE_REPO is $GIT_BUILD_DIR.
> warning: This will work, but may not be a sufficiently large repo
> warning: for representative measurements.
> not ok 1 - repack
> #	
> #		git repack -ad &&
> #		PACK=$(ls .git/objects/pack/*.pack | head -n1) &&
> #		test -f "$PACK" &&
> #		export PACK
> #	
> cannot open test-results/p5302-pack-index.subtests: No such file or directory at ./aggregate.perl line 153.

Implies that we're trying to _write_ to it, and that the problem is that
test-results doesn't exist. That should be set up by this part of
perf-lib:

  perf_results_dir=$TEST_OUTPUT_DIRECTORY/test-results
  test -n "$GIT_PERF_SUBSECTION" && perf_results_dir="$perf_results_dir/$GIT_PERF_SUBSECTION"
  mkdir -p "$perf_results_dir"
  rm -f "$perf_results_dir"/$(basename "$0" .sh).subtests

I don't know why that would be failing for you (and not producing any
kind of message!). You might need to poke at:

  ./p5302-pack-index.sh -x

to see what is going on.

It's also weird that the repack fails for you. And that (presumably?)
this is the only script that fails, since they would all want to do
this.

There are some bits of test-lib.sh that re-exec the script, but I think
we ironed out the weirdness there (and I use "--verbose-log" myself,
which is one such option). But just in case: do you set GIT_TEST_OPTS in
your config.mak?

-Peff

Re: Missing file in 2.23 (p5302-pack-index.subtests)?

[Theodore Y. Ts'o]

On Mon, Aug 26, 2019 at 10:27:00PM -0400, Jeff King wrote:
> > cannot open test-results/p5302-pack-index.subtests: No such file or directory at ./aggregate.perl line 153.
> 
> Implies that we're trying to _write_ to it, and that the problem is that
> test-results doesn't exist. That should be set up by this part of
> perf-lib:

Hmm....   test-results does exist after the failure:

<tytso@lambda> {/usr/projects/git/git-core/t/perf}   (master)
1174% rm -rf test-results ; ./run p5302-pack-index.sh
=== Running 1 tests in this tree ===
warning: $GIT_PERF_LARGE_REPO is $GIT_BUILD_DIR.
warning: This will work, but may not be a sufficiently large repo
warning: for representative measurements.
not ok 1 - repack
#	
#		git repack -ad &&
#		PACK=$(ls .git/objects/pack/*.pack | head -n1) &&
#		test -f "$PACK" &&
#		export PACK
#	
cannot open test-results/p5302-pack-index.subtests: No such file or directory at ./aggregate.perl line 153.
<tytso@lambda> {/usr/projects/git/git-core/t/perf}   (master)
1175% ls test-results
total 0
0 run_subsections.names

Maybe we're not in the right cwd for some reason?!?


> There are some bits of test-lib.sh that re-exec the script, but I think
> we ironed out the weirdness there (and I use "--verbose-log" myself,
> which is one such option). But just in case: do you set GIT_TEST_OPTS in
> your config.mak?

You mean in my top-level directory?  I don't have a config.mak file in
there:

<tytso@lambda> {/usr/projects/git/git-core}   (master)
1151% grep GIT_TEST_OPTS config.mak.*
<tytso@lambda> {/usr/projects/git/git-core}   (master)
1181% ls config.mak*
 4 config.mak.autogen   4 config.mak.dev   4 config.mak.in  24 config.mak.uname
<tytso@lambda> {/usr/projects/git/git-core}   (master)
1182% grep GIT_TEST_OPTS config.mak.*

					- Ted

Re: Missing file in 2.23 (p5302-pack-index.subtests)?

[Jeff King]

On Mon, Aug 26, 2019 at 10:58:11PM -0400, Theodore Y. Ts'o wrote:

> On Mon, Aug 26, 2019 at 10:27:00PM -0400, Jeff King wrote:
> > > cannot open test-results/p5302-pack-index.subtests: No such file or directory at ./aggregate.perl line 153.
> > 
> > Implies that we're trying to _write_ to it, and that the problem is that
> > test-results doesn't exist. That should be set up by this part of
> > perf-lib:
> 
> Hmm....   test-results does exist after the failure:

Ah, never mind. I was thinking it was coming from this line in perf-lib:

   echo "$test_count" >>"$perf_results_dir"/$base.subtests

But it it is pretty clear from this line of your output:

> cannot open test-results/p5302-pack-index.subtests: No such file or directory at ./aggregate.perl line 153.

...that this is the aggregate script that is run afterwards complaining.
So for whatever reason, the actual p5302 script is exiting early, before
it writes anything into the subtests file (from the line above).

That subtests write is done in test_wrapper_(), which is only triggered
for actual timing tests, not for setup tests (like the repack step that
starts this script). So the plausible sequence of events here is:

  1. The first "test_expect_success repack" test fails for some reason.
     Try running "./p5302-pack-index -v -i -x" to see more output.

  2. After the initial failure, the script exits totally rather than
     continue. That's due to this line in perf-lib, which acts as if
     "-i" is always set:

       # Performance tests should never fail.  If they do, stop immediately
       immediate=t

     That makes sense, since any timings we do after a setup step fails
     would invalidate the result. And if it's not the _first_ such step
     that fails, we'll just have a truncated subtests file, and some
     timings will be missing. But when the first one fails, there's no
     file at all.

     We could probably leave a more consistent state in that case. E.g.,
     something like this:

diff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh
index b58a43ea43..cfac2c3cbe 100644
--- a/t/perf/perf-lib.sh
+++ b/t/perf/perf-lib.sh
@@ -48,7 +48,7 @@ export MODERN_GIT
 perf_results_dir=$TEST_OUTPUT_DIRECTORY/test-results
 test -n "$GIT_PERF_SUBSECTION" && perf_results_dir="$perf_results_dir/$GIT_PERF_SUBSECTION"
 mkdir -p "$perf_results_dir"
-rm -f "$perf_results_dir"/$(basename "$0" .sh).subtests
+>"$perf_results_dir"/$(basename "$0" .sh).subtests
 
 die_if_build_dir_not_repo () {
 	if ! ( cd "$TEST_DIRECTORY/.." &&


     which would just quietly omit the rest of p5302. It is unfortunate
     that you wouldn't get some note in the output saying "hey, we
     didn't run some tests!".

     A lot of this has to do with the hackish way that the list of tests
     is generated. If you do something like:

       ./run HEAD^ HEAD p5302-pack-index.sh

     and the tests fail in HEAD^ but not HEAD, you'll get a nice table
     listing all of the tests, with a nil field for each one that didn't
     run in HEAD^.

     But that's because the second run, on HEAD, actually generated a
     correct p5302-pack-index.subtests file, that actually mentioned the
     tests! If the order were reversed, you'd get an empty list.

     So I think in the long run, it would be nice to have some way of
     generating the list that's more robust to failures. But I suspect
     doing that is going to be hard; a lot of this is rooted in the fact
     that there's no data structure with the set of tests, but literally
     an executable shell script that decides what to run.

So short answer: use "-v" to figure out why the repack is failing, and
the rest is just Git's perf suite being a bit hacky.

-Peff