Re: Git self test failure on Solaris 11.3

Re: Git self test failure on Solaris 11.3

[Eric Sunshine]

[forwarding to the Git list]

On Sun, Jun 2, 2019 at 6:23 AM Jeffrey Walton <noloader@gmail.com> wrote:
> On Sun, Jun 2, 2019 at 5:09 AM Jeffrey Walton <noloader@gmail.com> wrote:
> > I'm catching a self test failure on Solaris 11.3. Git 2.21 from sources.
> >
> > ok 8 - check normal git operations: two packs
> > ok 9 - add more packs
> > ok 10 - check normal git operations: mixed mode (two packs + extra)
> > ok 11 - write midx with twelve packs
> > not ok 12 - check normal git operations: twelve packs
> > #
> > #                       midx_git_two_modes "rev-list --objects --all" &&
> > #                       midx_git_two_modes "log --raw" &&
> > #                       midx_git_two_modes "count-objects --verbose" &&
> > #                       midx_git_two_modes "cat-file
> > --batch-all-objects --buffer --batch-check" &&
> > #                       midx_git_two_modes "cat-file
> > --batch-all-objects --buffer --batch-check --unsorted" sorted
> > #
> > ok 13 - verify multi-pack-index success
> > ok 14 - verify bad signature
> > ok 15 - verify bad version
> > ...
>
> Ick, this is bad. Intermittent failure. The second time around it passed.

Jeff Walton reported this to me privately. I'm not familiar with this
code and don't have time presently to investigate it, so I'm
forwarding it to the list in the hope that someone who knows the code
intimately (Stolee) or someone who has touched this code (Peff) might
have some idea about it. The failure is intermittent, so perhaps the
output of some command is not stable sort-wise(?).

Re: Git self test failure on Solaris 11.3

[Jeff King]

On Thu, Jun 06, 2019 at 01:18:01PM -0400, Eric Sunshine wrote:

> > > not ok 12 - check normal git operations: twelve packs
> > > #
> > > #                       midx_git_two_modes "rev-list --objects --all" &&
> > > #                       midx_git_two_modes "log --raw" &&
> > > #                       midx_git_two_modes "count-objects --verbose" &&
> > > #                       midx_git_two_modes "cat-file
> > > --batch-all-objects --buffer --batch-check" &&
> > > #                       midx_git_two_modes "cat-file
> > > --batch-all-objects --buffer --batch-check --unsorted" sorted
> > > #
> [...]
> 
> Jeff Walton reported this to me privately. I'm not familiar with this
> code and don't have time presently to investigate it, so I'm
> forwarding it to the list in the hope that someone who knows the code
> intimately (Stolee) or someone who has touched this code (Peff) might
> have some idea about it. The failure is intermittent, so perhaps the
> output of some command is not stable sort-wise(?).

I can't seem to get it to fail for me, even under high load. Do you know
which of those commands is failing (i.e., what does "-v -x" say)?

If it's the rev-list or log output, it could be due to traversal order.
I notice that the commit creation doesn't increment the timestamps. This
might help:

diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh
index 70926b5bc0..79aa97efce 100755
--- a/t/t5319-multi-pack-index.sh
+++ b/t/t5319-multi-pack-index.sh
@@ -60,6 +60,7 @@ commit_and_list_objects () {
 	} >file_101 &&
 	git update-index --add file_101 &&
 	tree=$(git write-tree) &&
+	test_tick &&
 	commit=$(git commit-tree $tree -p HEAD</dev/null) &&
 	{
 		echo $tree &&

Though I think in each case that the results should be deterministic for
a given set of commits (and the tests here are just running over the
same set of commits with and without midx; so we should see the same
output in both cases).

The normal "cat-file" output should be sorted by sha1, so should be
stable. The unsorted output will be in whatever order we happen to hit
the packfiles, which _would_ be subject to timestamp variations.
However, Jeffrey's email mentions Git 2.21, which predates my b4a14394af
(t5319: fix bogus cat-file argument, 2019-04-05). So before that,
"cat-file --unsorted" should always be producing no output. :)

After the switch to correctly use --unordered, I could believe there
might be a timestamp issue, but the "sorted" flag there means that we
should be sorting the output before comparing.

I can't reproduce the intermittent failure either on 2.21.0, or with
v2.22.0-rc3.

-Peff

Re: Git self test failure on Solaris 11.3

[Eric Sunshine]

On Thu, Jun 6, 2019 at 1:35 PM Jeff King <peff@peff.net> wrote:
> On Thu, Jun 06, 2019 at 01:18:01PM -0400, Eric Sunshine wrote:
> > > > not ok 12 - check normal git operations: twelve packs
> >
> > Jeff Walton reported this to me privately. I'm not familiar with this
> > code and don't have time presently to investigate it, so I'm
> > forwarding it to the list in the hope that someone who knows the code
> > intimately (Stolee) or someone who has touched this code (Peff) might
> > have some idea about it. The failure is intermittent, so perhaps the
> > output of some command is not stable sort-wise(?).
>
> I can't seem to get it to fail for me, even under high load. Do you know
> which of those commands is failing (i.e., what does "-v -x" say)?

I don't know.

> I can't reproduce the intermittent failure either on 2.21.0, or with
> v2.22.0-rc3.

I can't reproduce it either on Jeff's Solaris box. Perhaps Jeff can
add "-v -x" to his automated build/test script in order to help
diagnose the problem if it occurs again.

Thanks for looking into it.

Re: Git self test failure on Solaris 11.3

[Jeff King]

On Thu, Jun 06, 2019 at 03:00:00PM -0400, Eric Sunshine wrote:

> > I can't reproduce the intermittent failure either on 2.21.0, or with
> > v2.22.0-rc3.
> 
> I can't reproduce it either on Jeff's Solaris box. Perhaps Jeff can
> add "-v -x" to his automated build/test script in order to help
> diagnose the problem if it occurs again.

Jeff: try "--verbose-log -x" if you're doing this as part of an automated
run. That will leave you nice logs in t/test-results without dumping it
all (and in particular, if you run the tests in parallel, it will avoid
interleaving output from various tests).

-Peff

Re: Git self test failure on Solaris 11.3

[Derrick Stolee]

On 6/6/2019 3:05 PM, Jeff King wrote:
> On Thu, Jun 06, 2019 at 03:00:00PM -0400, Eric Sunshine wrote:
> 
>>> I can't reproduce the intermittent failure either on 2.21.0, or with
>>> v2.22.0-rc3.
>>
>> I can't reproduce it either on Jeff's Solaris box. Perhaps Jeff can
>> add "-v -x" to his automated build/test script in order to help
>> diagnose the problem if it occurs again.
> 
> Jeff: try "--verbose-log -x" if you're doing this as part of an automated
> run. That will leave you nice logs in t/test-results without dumping it
> all (and in particular, if you run the tests in parallel, it will avoid
> interleaving output from various tests).

Please keep me posted with any updates with exactly which command failed.
These comparison tests are intended to catch surprising interactions, but
are not very narrow. We'll need to know the exact failure spot, and the
failure could take a few forms:

* The git command failed with an error.

* The midx-enabled and midx-disabled commands differ in output.

Between these two options and the different commands that could have failed,
it's hard to dig in without more to go on.

Thanks,
-Stolee