Loss of vDSO support on s390x

Loss of vDSO support on s390x

[Florian Weimer]

time, gettimeofday, and clock_gettime always seem to make system calls
now.

Is this intentional?  I'm worried that this is a major performance
regression.

Thanks,
Florian

Re: Loss of vDSO support on s390x

[Siddhesh Poyarekar]

On 15/01/20 3:20 pm, Florian Weimer wrote:
> time, gettimeofday, and clock_gettime always seem to make system calls
> now.
> 
> Is this intentional?  I'm worried that this is a major performance
> regression.

+Adhemerval; this may be related to the vDSO patchset.

Siddhesh

Re: Loss of vDSO support on s390x

[Florian Weimer]

* Siddhesh Poyarekar:

> On 15/01/20 3:20 pm, Florian Weimer wrote:
>> time, gettimeofday, and clock_gettime always seem to make system calls
>> now.
>> 
>> Is this intentional?  I'm worried that this is a major performance
>> regression.
>
> +Adhemerval; this may be related to the vDSO patchset.

The kernel I used disables the vDSO with an explicit loader invocation.
I assume this was fixed with this commit:

commit 404861e15b5fa7edbab22400f9174c1a21fde731
Author: Heiko Carstens <heiko.carstens@de.ibm.com>
Date:   Wed Aug 7 14:31:59 2019 +0200

    s390/vdso: map vdso also for statically linked binaries
    
    s390 does not map the vdso for statically linked binaries, assuming
    that this doesn't make sense. See commit fc5243d98ac2 ("[S390]
    arch_setup_additional_pages arguments").
    
    However with glibc commit d665367f596d ("linux: Enable vDSO for static
    linking as default (BZ#19767)") and commit 5e855c895401 ("s390: Enable
    VDSO for static linking") the vdso is also used for statically linked
    binaries - if the kernel would make it available.
    
    Therefore map the vdso always, just like all other architectures.

If I fix this, by hardcoding the new loader, I only see a system call
for the time emulation.  It looks like the vDSO does not have
acceleration for CLOCK_REALTIME_COARSE, which is used by the time
emulation.

I still think this needs to be fixed.

Thanks,
Florian

Re: Loss of vDSO support on s390x

[Stefan Liebler]

On 1/15/20 1:37 PM, Florian Weimer wrote:
> * Siddhesh Poyarekar:
> 
>> On 15/01/20 3:20 pm, Florian Weimer wrote:
>>> time, gettimeofday, and clock_gettime always seem to make system calls
>>> now.
>>>
>>> Is this intentional?  I'm worried that this is a major performance
>>> regression. >>
>> +Adhemerval; this may be related to the vDSO patchset.
> 
> The kernel I used disables the vDSO with an explicit loader invocation.
> I assume this was fixed with this commit:
> 
> commit 404861e15b5fa7edbab22400f9174c1a21fde731
> Author: Heiko Carstens <heiko.carstens@de.ibm.com>
> Date:   Wed Aug 7 14:31:59 2019 +0200
> 
>      s390/vdso: map vdso also for statically linked binaries
>      
>      s390 does not map the vdso for statically linked binaries, assuming
>      that this doesn't make sense. See commit fc5243d98ac2 ("[S390]
>      arch_setup_additional_pages arguments").
>      
>      However with glibc commit d665367f596d ("linux: Enable vDSO for static
>      linking as default (BZ#19767)") and commit 5e855c895401 ("s390: Enable
>      VDSO for static linking") the vdso is also used for statically linked
>      binaries - if the kernel would make it available.
>      
>      Therefore map the vdso always, just like all other architectures.
> 
> If I fix this, by hardcoding the new loader, I only see a system call
> for the time emulation.  It looks like the vDSO does not have
> acceleration for CLOCK_REALTIME_COARSE, which is used by the time
> emulation.
According to <kernel>/arch/s390/kernel/vdso64/clock_gettime.S there is 
support for:
- CLOCK_MONOTONIC
- CLOCK_MONOTONIC_COARSE
- CLOCK_REALTIME_COARSE
- CLOCK_REALTIME
- CPUCLOCK_VIRT for this thread
(See 
https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/tree/arch/s390/kernel/vdso64/clock_gettime.S?id=2115fbf7210bd053ba55a95e7ebc366df41aa9cf)

> 
> I still think this needs to be fixed.
> 
> Thanks,
> Florian
> 

Do you run on s390x (64bit) or s390 (31bit)?
Heiko has also removed the vdso for 31bit in linux 5.5:
"s390: remove compat vdso code"
https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/commit/?id=2115fbf7210bd053ba55a95e7ebc366df41aa9cf

Please double check if the vdso is mapped and the address is passed via 
AT_SYSINFO_EHDR. I've just run a test program linked against current 
glibc and stepped to the vdso.

But on 31bit, the vdso is mapped, but I end in a syscall as my used 
kernel has support for __NR_clock_gettime64 but has no support for 
HAVE_CLOCK_GETTIME64_VSYSCALL (only for HAVE_CLOCK_GETTIME_VSYSCALL).

Bye,
Stefan

Re: Loss of vDSO support on s390x

[Florian Weimer]

* Stefan Liebler:

> According to <kernel>/arch/s390/kernel/vdso64/clock_gettime.S there is
> support for:
> - CLOCK_MONOTONIC
> - CLOCK_MONOTONIC_COARSE
> - CLOCK_REALTIME_COARSE
> - CLOCK_REALTIME
> - CPUCLOCK_VIRT for this thread
> (See
> https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/tree/arch/s390/kernel/vdso64/clock_gettime.S?id=2115fbf7210bd053ba55a95e7ebc366df41aa9cf)

Hmm.  Looks like I'm missing this kernel commit:

commit b7eacb59cd7fb5e98852186e485c0c865f862645
Author: Martin Schwidefsky <schwidefsky@de.ibm.com>
Date:   Fri Aug 29 12:31:45 2014 +0200

    s390/vdso: add vdso support for coarse clocks
    
    Add CLOCK_REALTIME_COARSE and CLOCK_MONOTONIC_COARSE optimization to
    the 64-bit and 31-bit vdso.

That went into 3.18, so maybe that's old enough.  It will impact people
who run newer containers on el7 hosts, though.  Maybe we should request
a backport of that commit?

> Do you run on s390x (64bit) or s390 (31bit)?

It's s390x.  I expect s390 to be worse (as you showed), but I'm not sure
if we need to fix it proactively.

Thanks,
Florian