Math errors

Math errors

[Alistair Francis]

Hey,

I'm running the `make check` tests on RV32 inside QEMU and about 800
math tests are failing with errors along these lines:

$ cat math/test-double-erf.out
Failure: Test: erf (0x1.4p+0)
Result:
 is:          9.2290012825645817e-01   0x1.d8865d98abe00p-1
 should be:   9.2290012825645829e-01   0x1.d8865d98abe01p-1
 difference:  1.1102230246251565e-16   0x1.0000000000000p-53
 ulp       :  1.0000
 max.ulp   :  0.0000
Maximal error of `erf'
 is      : 1 ulp
 accepted: 0 ulp
Failure: Test: erf_downward (-0x1.4d32f4p-12)
Result:
 is:         -3.5855754629430663e-04  -0x1.77f98ef609eb3p-12
 should be:  -3.5855754629430669e-04  -0x1.77f98ef609eb4p-12
 difference:  5.4210108624275221e-20   0x1.0000000000000p-64
 ulp       :  1.0000
 max.ulp   :  0.0000
Failure: Test: erf_downward (0x1.44e722p+0)
Result:
 is:          9.2732266856644440e-01   0x1.daca096caa26ap-1
 should be:   9.2732266856644429e-01   0x1.daca096caa269p-1
 difference:  1.1102230246251565e-16   0x1.0000000000000p-53

It seems the tests are failing due to small differences in the
floating point values.

Has anyone seen this before? I don't see how this can be related
specifically to the RV32 port, but I didn't see these errors when
running the ARM 32-bit testing. It's possible we are loosing
information in the QEMU floating point implementation for RV32.

Alistair

Re: Math errors

[Joseph Myers]

On Mon, 30 Dec 2019, Alistair Francis wrote:

> Hey,
> 
> I'm running the `make check` tests on RV32 inside QEMU and about 800
> math tests are failing with errors along these lines:

You need to generate libm-test-ulps for RV32 ("make regen-ulps") (also 
remember to create a corresponding libm-test-ulps-name file for use in 
generating the table in the manual).

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Math errors

[Joseph Myers]

On Mon, 30 Dec 2019, Joseph Myers wrote:

> On Mon, 30 Dec 2019, Alistair Francis wrote:
> 
> > Hey,
> > 
> > I'm running the `make check` tests on RV32 inside QEMU and about 800
> > math tests are failing with errors along these lines:
> 
> You need to generate libm-test-ulps for RV32 ("make regen-ulps") (also 
> remember to create a corresponding libm-test-ulps-name file for use in 
> generating the table in the manual).

I should add: first try renaming the RV64 libm-test-ulps to make it 
generic for both RV32 and RV64.  You should only need a separate file for 
RV32 if there is a clear reason for different results between the two.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Math errors

[Alistair Francis]

On Mon, Dec 30, 2019 at 2:29 PM Joseph Myers <joseph@codesourcery.com> wrote:
>
> On Mon, 30 Dec 2019, Joseph Myers wrote:
>
> > On Mon, 30 Dec 2019, Alistair Francis wrote:
> >
> > > Hey,
> > >
> > > I'm running the `make check` tests on RV32 inside QEMU and about 800
> > > math tests are failing with errors along these lines:
> >
> > You need to generate libm-test-ulps for RV32 ("make regen-ulps") (also
> > remember to create a corresponding libm-test-ulps-name file for use in
> > generating the table in the manual).
>
> I should add: first try renaming the RV64 libm-test-ulps to make it
> generic for both RV32 and RV64.  You should only need a separate file for
> RV32 if there is a clear reason for different results between the two.

Thanks Joseph,

I moved the RV64 one to be generic and that seems to fix most of the
math tests. I tried `make regen-ulps` after that, just to be sure, and
it didn't generate a diff.

There are a handful of math tests that seem to fail, but if I manually
run the test a few times I will eventually get a pass. Any hints on
what to do there?

Alistair

>
> --
> Joseph S. Myers
> joseph@codesourcery.com

Re: Math errors

[Szabolcs Nagy]

On 03/01/2020 02:18, Alistair Francis wrote:
> On Mon, Dec 30, 2019 at 2:29 PM Joseph Myers <joseph@codesourcery.com> wrote:
>>
>> On Mon, 30 Dec 2019, Joseph Myers wrote:
>>
>>> On Mon, 30 Dec 2019, Alistair Francis wrote:
>>>
>>>> Hey,
>>>>
>>>> I'm running the `make check` tests on RV32 inside QEMU and about 800
>>>> math tests are failing with errors along these lines:
>>>
>>> You need to generate libm-test-ulps for RV32 ("make regen-ulps") (also
>>> remember to create a corresponding libm-test-ulps-name file for use in
>>> generating the table in the manual).
>>
>> I should add: first try renaming the RV64 libm-test-ulps to make it
>> generic for both RV32 and RV64.  You should only need a separate file for
>> RV32 if there is a clear reason for different results between the two.
> 
> Thanks Joseph,
> 
> I moved the RV64 one to be generic and that seems to fix most of the
> math tests. I tried `make regen-ulps` after that, just to be sure, and
> it didn't generate a diff.

regen-ulps should create a math/NewUlps file.
(details are in math/README.libm-test)

> 
> There are a handful of math tests that seem to fail, but if I manually
> run the test a few times I will eventually get a pass. Any hints on
> what to do there?

flaky math test is not normal, you need to investigate.

> 
> Alistair
> 
>>
>> --
>> Joseph S. Myers
>> joseph@codesourcery.com

Re: Math errors

[Alistair Francis]

On Fri, Jan 3, 2020 at 1:30 AM Szabolcs Nagy <Szabolcs.Nagy@arm.com> wrote:
>
> On 03/01/2020 02:18, Alistair Francis wrote:
> > On Mon, Dec 30, 2019 at 2:29 PM Joseph Myers <joseph@codesourcery.com> wrote:
> >>
> >> On Mon, 30 Dec 2019, Joseph Myers wrote:
> >>
> >>> On Mon, 30 Dec 2019, Alistair Francis wrote:
> >>>
> >>>> Hey,
> >>>>
> >>>> I'm running the `make check` tests on RV32 inside QEMU and about 800
> >>>> math tests are failing with errors along these lines:
> >>>
> >>> You need to generate libm-test-ulps for RV32 ("make regen-ulps") (also
> >>> remember to create a corresponding libm-test-ulps-name file for use in
> >>> generating the table in the manual).
> >>
> >> I should add: first try renaming the RV64 libm-test-ulps to make it
> >> generic for both RV32 and RV64.  You should only need a separate file for
> >> RV32 if there is a clear reason for different results between the two.
> >
> > Thanks Joseph,
> >
> > I moved the RV64 one to be generic and that seems to fix most of the
> > math tests. I tried `make regen-ulps` after that, just to be sure, and
> > it didn't generate a diff.
>
> regen-ulps should create a math/NewUlps file.
> (details are in math/README.libm-test)

Yep, I just meant that I diffed that to the RV64 one.

>
> >
> > There are a handful of math tests that seem to fail, but if I manually
> > run the test a few times I will eventually get a pass. Any hints on
> > what to do there?
>
> flaky math test is not normal, you need to investigate.

It turns out that there is a difference between libm-test-ulps for
RV32 and RV64. The gen-libm-test.py script was failing to run as I
don't have Python2 installed, so the file wasn't being updated. If I
manually run the script with Python3 I can generate a libm-test-ulps
file that is somewhat different to the RV64 one.

I'm running the tests now, but hopefully that fixes the failing math test cases.

Alistair

>
> >
> > Alistair
> >
> >>
> >> --
> >> Joseph S. Myers
> >> joseph@codesourcery.com
>

Re: Math errors

[Jim Wilson]

On Thu, Jan 2, 2020 at 6:18 PM Alistair Francis <alistair23@gmail.com> wrote:
> There are a handful of math tests that seem to fail, but if I manually
> run the test a few times I will eventually get a pass. Any hints on
> what to do there?

There is a known bug in the RISC-V linux port where FP regs can leak
across fork and exec.  This is normally not a fatal problem, except
for some tests that assume that the FP exception flags are clear on
program start.  This can be false if they were set in the parent
process, which can cause some glibc math test failures.  This can
result in failures that appear/disappear when run manually versus from
the testsuite driver.  There are only about 5 tests that are affected.
The bug was finally fixed about a year and a half after I first
reported it.  That happened only a few months ago.  If you don't have
a recent enough version of the linux kernel you may not have the fix.

Jim

Re: Math errors

[Alistair Francis]

On Mon, Jan 6, 2020 at 5:29 PM Jim Wilson <jimw@sifive.com> wrote:
>
> On Thu, Jan 2, 2020 at 6:18 PM Alistair Francis <alistair23@gmail.com> wrote:
> > There are a handful of math tests that seem to fail, but if I manually
> > run the test a few times I will eventually get a pass. Any hints on
> > what to do there?
>
> There is a known bug in the RISC-V linux port where FP regs can leak
> across fork and exec.  This is normally not a fatal problem, except
> for some tests that assume that the FP exception flags are clear on
> program start.  This can be false if they were set in the parent
> process, which can cause some glibc math test failures.  This can
> result in failures that appear/disappear when run manually versus from
> the testsuite driver.  There are only about 5 tests that are affected.
> The bug was finally fixed about a year and a half after I first
> reported it.  That happened only a few months ago.  If you don't have
> a recent enough version of the linux kernel you may not have the fix.

Thanks for that Jim.

I have the latest 5.4.x kernel, I'm not sure if that has it or not.

I see a lot of math failures, so I'm not sure if it's the same issue.

Right now I have all of these failing:

    FAIL: math/atest-exp
    FAIL: math/atest-exp2
    FAIL: math/atest-sincos
    FAIL: math/test-double-cacos
    FAIL: math/test-double-cosh
    FAIL: math/test-double-fromfpx
    FAIL: math/test-double-islessgreater
    FAIL: math/test-double-iszero
    FAIL: math/test-double-jn
    FAIL: math/test-double-llrint
    FAIL: math/test-double-log1p
    FAIL: math/test-double-ufromfp
    UNSUPPORTED: math/test-fesetexcept-traps
    UNSUPPORTED: math/test-fexcept-traps
    FAIL: math/test-float-cabs
    FAIL: math/test-float-ilogb
    FAIL: math/test-float-isnormal
    FAIL: math/test-float-ldouble-div
    FAIL: math/test-float128-fmod
    FAIL: math/test-float128-iseqsig
    FAIL: math/test-float128-ufromfpx
    FAIL: math/test-float32-catan
    FAIL: math/test-float32-csinh
    FAIL: math/test-float32-float64x-add
    FAIL: math/test-float32-iszero
    FAIL: math/test-float32-jn
    FAIL: math/test-float32-significand
    FAIL: math/test-float32-sincos
    FAIL: math/test-float32-tanh
    FAIL: math/test-float32-yn
    FAIL: math/test-float32x-asinh
    FAIL: math/test-float32x-exp
    FAIL: math/test-float32x-log10
    FAIL: math/test-float32x-yn
    FAIL: math/test-float64-asinh
    FAIL: math/test-float64-canonicalize
    FAIL: math/test-float64-getpayload
    FAIL: math/test-float64-j1
    FAIL: math/test-float64-jn
    FAIL: math/test-float64x-cexp
    FAIL: math/test-float64x-ctanh
    FAIL: math/test-idouble-acos
    FAIL: math/test-idouble-catanh
    FAIL: math/test-idouble-copysign
    FAIL: math/test-idouble-csinh
    FAIL: math/test-idouble-erf
    FAIL: math/test-idouble-fminmag
    FAIL: math/test-idouble-hypot
    FAIL: math/test-idouble-lgamma
    FAIL: math/test-idouble-log1p
    FAIL: math/test-idouble-setpayload
    FAIL: math/test-ifloat-cacos
    FAIL: math/test-ifloat-cacosh
    FAIL: math/test-ifloat-casin
    FAIL: math/test-ifloat-erf
    FAIL: math/test-ifloat-llround
    FAIL: math/test-ifloat-pow
    FAIL: math/test-ifloat-setpayload
    FAIL: math/test-ifloat-sincos
    FAIL: math/test-ifloat-tan
    FAIL: math/test-ifloat-tgamma
    FAIL: math/test-ifloat128-ctan
    FAIL: math/test-ifloat128-j0
    FAIL: math/test-ifloat32-clog10
    FAIL: math/test-ifloat32-issignaling
    FAIL: math/test-ifloat32x-cpow
    FAIL: math/test-ifloat32x-fmaxmag
    FAIL: math/test-ifloat32x-j0
    FAIL: math/test-ifloat32x-remainder
    FAIL: math/test-ifloat64-cexp
    FAIL: math/test-ifloat64-floor
    FAIL: math/test-ifloat64-fmod
    FAIL: math/test-ifloat64-lrint
    FAIL: math/test-ifloat64-sincos
    FAIL: math/test-ildouble-lgamma
    FAIL: math/test-ldouble-exp
    FAIL: math/test-ldouble-fmod
    FAIL: math/test-ldouble-ilogb
    UNSUPPORTED: math/test-matherr
    UNSUPPORTED: math/test-matherr-2
    UNSUPPORTED: math/test-nearbyint-except-2

Alistair

>
> Jim

Re: Math errors

[Jim Wilson]

On Mon, Jan 6, 2020 at 5:33 PM Alistair Francis <alistair23@gmail.com> wrote:
> I have the latest 5.4.x kernel, I'm not sure if that has it or not.

I don't know.  I don't follow linux kernel development that closely.
But the failures I sometimes see are test-fenv and test-fpucw which I
don't see in your list.

I only run glibc testsuite on hardware though, and I've only been
testing rv64gc support.  I haven't tried qemu.  In the RISC-V software
meeting this morning, Palmer mentioned that there are some known
RISC-V qemu FP bugs, for instance fcmp doesn't set the FP reg dirty
bit when an exception flag is set, which could cause a testcase
failure if a context switch happens at the wrong time.  This could
cause tests to sometimes fail and sometimes work.  I have seen some
RISC-V qemu FP bugs myself, for instance with NaN-boxing, where
sometimes single float outputs aren't NaN-boxed, and sometimes
non-NaN-boxed single float inputs don't cause failures.  This is a
problem I noticed by accident, I don't know if that has been fixed
yet, and I don't know if that would cause glibc testsuite failures.
Someone will have to look at the rv32gc math failures, and keep an
open mind that they could be glibc, qemu, gcc, linux kernel, etc
problems.  That of course assumes you have an up-to-date ulps file as
Joseph already mentioned.

Jim

Re: Math errors

[Alistair Francis]

On Wed, Jan 8, 2020 at 4:29 AM Jim Wilson <jimw@sifive.com> wrote:
>
> On Mon, Jan 6, 2020 at 5:33 PM Alistair Francis <alistair23@gmail.com> wrote:
> > I have the latest 5.4.x kernel, I'm not sure if that has it or not.
>
> I don't know.  I don't follow linux kernel development that closely.
> But the failures I sometimes see are test-fenv and test-fpucw which I
> don't see in your list.
>
> I only run glibc testsuite on hardware though, and I've only been
> testing rv64gc support.  I haven't tried qemu.  In the RISC-V software
> meeting this morning, Palmer mentioned that there are some known
> RISC-V qemu FP bugs, for instance fcmp doesn't set the FP reg dirty
> bit when an exception flag is set, which could cause a testcase
> failure if a context switch happens at the wrong time.  This could

I have noticed that running the tests not in parallel gives much better results

> cause tests to sometimes fail and sometimes work.  I have seen some
> RISC-V qemu FP bugs myself, for instance with NaN-boxing, where
> sometimes single float outputs aren't NaN-boxed, and sometimes
> non-NaN-boxed single float inputs don't cause failures.  This is a
> problem I noticed by accident, I don't know if that has been fixed
> yet, and I don't know if that would cause glibc testsuite failures.
> Someone will have to look at the rv32gc math failures, and keep an
> open mind that they could be glibc, qemu, gcc, linux kernel, etc

Yeah, that's what I was thinking. There are only a few math tests left
failing, so it's not too bad. Thanks for the information Jim.

Alistair

> problems.  That of course assumes you have an up-to-date ulps file as
> Joseph already mentioned.
>
> Jim

Re: Math errors

[Jonathan Nieder]

Hi,

Alistair Francis wrote:
> On Wed, Jan 8, 2020 at 4:29 AM Jim Wilson <jimw@sifive.com> wrote:
>> On Mon, Jan 6, 2020 at 5:33 PM Alistair Francis <alistair23@gmail.com> wrote:

>>> I have the latest 5.4.x kernel, I'm not sure if that has it or not.
>>
>> I don't know.  I don't follow linux kernel development that closely.
>> But the failures I sometimes see are test-fenv and test-fpucw which I
>> don't see in your list.
>>
>> I only run glibc testsuite on hardware though, and I've only been
>> testing rv64gc support.  I haven't tried qemu.  In the RISC-V software
>> meeting this morning, Palmer mentioned that there are some known
>> RISC-V qemu FP bugs, for instance fcmp doesn't set the FP reg dirty
>> bit when an exception flag is set, which could cause a testcase
>> failure if a context switch happens at the wrong time.  This could
>
> I have noticed that running the tests not in parallel gives much better results

Interesting.  Can you provide full "uname -r" output?  What Linux
distribution do you use?

I suspect the issue Jim mentioned is fixed by v5.3-rc5~12^2~1 ("riscv:
Correct the initialized flow of FP register", 2019-08-14), which you
would already have.  That said, the 5.5-rc series has some more fp reg
handling fixes.

[...]
>> Someone will have to look at the rv32gc math failures, and keep an
>> open mind that they could be glibc, qemu, gcc, linux kernel, etc
>
> Yeah, that's what I was thinking. There are only a few math tests left
> failing, so it's not too bad. Thanks for the information Jim.

That it appears more often with threading does suggest some kind of FP
register leakage.

Thanks and happy hunting,
Jonathan

Re: Math errors

[Alistair Francis]

On Tue, Jan 14, 2020 at 12:28 PM Jonathan Nieder <jrnieder@gmail.com> wrote:
>
> Hi,
>
> Alistair Francis wrote:
> > On Wed, Jan 8, 2020 at 4:29 AM Jim Wilson <jimw@sifive.com> wrote:
> >> On Mon, Jan 6, 2020 at 5:33 PM Alistair Francis <alistair23@gmail.com> wrote:
>
> >>> I have the latest 5.4.x kernel, I'm not sure if that has it or not.
> >>
> >> I don't know.  I don't follow linux kernel development that closely.
> >> But the failures I sometimes see are test-fenv and test-fpucw which I
> >> don't see in your list.
> >>
> >> I only run glibc testsuite on hardware though, and I've only been
> >> testing rv64gc support.  I haven't tried qemu.  In the RISC-V software
> >> meeting this morning, Palmer mentioned that there are some known
> >> RISC-V qemu FP bugs, for instance fcmp doesn't set the FP reg dirty
> >> bit when an exception flag is set, which could cause a testcase
> >> failure if a context switch happens at the wrong time.  This could
> >
> > I have noticed that running the tests not in parallel gives much better results
>
> Interesting.  Can you provide full "uname -r" output?  What Linux
> distribution do you use?

I don't have it in front of me right now, but it's 5.4.x and built
from OpenEmbedded.

>
> I suspect the issue Jim mentioned is fixed by v5.3-rc5~12^2~1 ("riscv:
> Correct the initialized flow of FP register", 2019-08-14), which you
> would already have.  That said, the 5.5-rc series has some more fp reg
> handling fixes.

It could be a QEMU issue:
https://lists.gnu.org/archive/html/qemu-devel/2020-01/msg02455.html

Alistair

>
> [...]
> >> Someone will have to look at the rv32gc math failures, and keep an
> >> open mind that they could be glibc, qemu, gcc, linux kernel, etc
> >
> > Yeah, that's what I was thinking. There are only a few math tests left
> > failing, so it's not too bad. Thanks for the information Jim.
>
> That it appears more often with threading does suggest some kind of FP
> register leakage.
>
> Thanks and happy hunting,
> Jonathan

Re: Math errors

[Alistair Francis]

On Tue, Jan 14, 2020 at 2:57 PM Alistair Francis <alistair23@gmail.com> wrote:
>
> On Tue, Jan 14, 2020 at 12:28 PM Jonathan Nieder <jrnieder@gmail.com> wrote:
> >
> > Hi,
> >
> > Alistair Francis wrote:
> > > On Wed, Jan 8, 2020 at 4:29 AM Jim Wilson <jimw@sifive.com> wrote:
> > >> On Mon, Jan 6, 2020 at 5:33 PM Alistair Francis <alistair23@gmail.com> wrote:
> >
> > >>> I have the latest 5.4.x kernel, I'm not sure if that has it or not.
> > >>
> > >> I don't know.  I don't follow linux kernel development that closely.
> > >> But the failures I sometimes see are test-fenv and test-fpucw which I
> > >> don't see in your list.
> > >>
> > >> I only run glibc testsuite on hardware though, and I've only been
> > >> testing rv64gc support.  I haven't tried qemu.  In the RISC-V software
> > >> meeting this morning, Palmer mentioned that there are some known
> > >> RISC-V qemu FP bugs, for instance fcmp doesn't set the FP reg dirty
> > >> bit when an exception flag is set, which could cause a testcase
> > >> failure if a context switch happens at the wrong time.  This could
> > >
> > > I have noticed that running the tests not in parallel gives much better results
> >
> > Interesting.  Can you provide full "uname -r" output?  What Linux
> > distribution do you use?
>
> I don't have it in front of me right now, but it's 5.4.x and built
> from OpenEmbedded.

# uname -r
5.4.7

Alistair

>
> >
> > I suspect the issue Jim mentioned is fixed by v5.3-rc5~12^2~1 ("riscv:
> > Correct the initialized flow of FP register", 2019-08-14), which you
> > would already have.  That said, the 5.5-rc series has some more fp reg
> > handling fixes.
>
> It could be a QEMU issue:
> https://lists.gnu.org/archive/html/qemu-devel/2020-01/msg02455.html
>
> Alistair
>
> >
> > [...]
> > >> Someone will have to look at the rv32gc math failures, and keep an
> > >> open mind that they could be glibc, qemu, gcc, linux kernel, etc
> > >
> > > Yeah, that's what I was thinking. There are only a few math tests left
> > > failing, so it's not too bad. Thanks for the information Jim.
> >
> > That it appears more often with threading does suggest some kind of FP
> > register leakage.
> >
> > Thanks and happy hunting,
> > Jonathan