From: Adhemerval Zanella via Libc-alpha <libc-alpha@sourceware.org>
To: libc-alpha@sourceware.org
Cc: Norbert Manthey <nmanthey@conp-solutions.com>,
Siddhesh Poyarekar <siddhesh@sourceware.org>,
Guillaume Morin <guillaume@morinfr.org>
Subject: [PATCH 0/3] malloc: improve THP effectiveness
Date: Fri, 13 Aug 2021 18:04:26 -0300 [thread overview]
Message-ID: <20210813210429.1147112-1-adhemerval.zanella@linaro.org> (raw)
Linux Transparent Huge Pages (THP) current support three different
states [1]: 'never', 'madvise', and 'always'. The 'never' is
self-explanatory and 'always' will enable THP for all anonymous
memory. However, 'madvise' is still the default for some system and
for such case THP will be only used if the memory range is explicity
advertise by the program through the madvise(MADV_HUGEPAGE) call.
This patchset adds a new tunable, 'glibc.malloc.thp_pagesize', which
allows the user to explicit use THP on anonymous page even if the
'madvise' state is set. The usage should be transparent for mmap()
call, and the madvise(MADV_HUGEPAGE) call is only set for sizes large
than the THP huge page size.
The sbrk() change alters the program memory allocation since the
increment is now aligned to the huge page size, instead of default page
size. It is enable as with the new tunable.
This patchset adds THP for aarch64, mips, powerpc, riscv, s390, sparc,
and x86. These are the architecture that have
HAVE_ARCH_TRANSPARENT_HUGEPAGE as default and define the internal
flags for THP support (I might have missed some architecture).
Although it does improve THP effectiveness, it does not provide the same
features from libhugetlsfs morecore implementation [2], since it does
use MAP_HUGETLB explicit on mmap. And I think this is not what we want
for glibc, it requires additional setup from the admin to mount the
hugetlsfs and reserve the pages with it outside from glibc scope.
The performance improvements are really dependent of the workload
and the platform, however a simple testcase might show the possible
improvements:
$ cat hugepages.cc
#include <unordered_map>
int
main (int argc, char *argv[])
{
std::size_t iters = 10000000;
std::unordered_map <std::size_t, std::size_t> ht;
ht.reserve (iters);
for (std::size_t i = 0; i < iters; ++i)
ht.try_emplace (i, i);
return 0;
}
$ g++ -std=c++17 -O2 hugepages.cc -o hugepages
On a x86_64 (Ryzen 9 5900X):
Performance counter stats for 'env
GLIBC_TUNABLES=glibc.malloc.thp_pagesize=0 ./testrun.sh ./hugepages':
98,874 faults
717,059 dTLB-loads
411,701 dTLB-load-misses # 57.42% of all dTLB
cache accesses
3,754,927 cache-misses # 8.479 % of all
cache refs
44,287,580 cache-references
0.315278378 seconds time elapsed
0.238635000 seconds user
0.076714000 seconds sys
Performance counter stats for 'env
GLIBC_TUNABLES=glibc.malloc.thp_pagesize=1 ./testrun.sh ./hugepages':
1,871 faults
120,035 dTLB-loads
19,882 dTLB-load-misses # 16.56% of all dTLB
cache accesses
4,182,942 cache-misses # 7.452 % of all
cache refs
56,128,995 cache-references
0.262620733 seconds time elapsed
0.222233000 seconds user
0.040333000 seconds sys
On an AArch64 (cortex A72):
Performance counter stats for 'env
GLIBC_TUNABLES=glibc.malloc.thp_pagesize=0 ./testrun.sh ./hugepages':
98835 faults
2007234756 dTLB-loads
4613669 dTLB-load-misses # 0.23% of all dTLB
cache accesses
8831801 cache-misses # 0.504 % of all
cache refs
1751391405 cache-references
0.616782575 seconds time elapsed
0.460946000 seconds user
0.154309000 seconds sys
Performance counter stats for 'env
GLIBC_TUNABLES=glibc.malloc.thp_pagesize=1 ./testrun.sh ./hugepages':
955 faults
1787401880 dTLB-loads
224034 dTLB-load-misses # 0.01% of all dTLB
cache accesses
5480917 cache-misses # 0.337 % of all
cache refs
1625937858 cache-references
0.487773443 seconds time elapsed
0.440894000 seconds user
0.046465000 seconds sys
And on a powerpc64 (POWER8):
Performance counter stats for 'env
GLIBC_TUNABLES=glibc.malloc.thp_pagesize=0 ./testrun.sh ./hugepages
':
5453 faults
9940 dTLB-load-misses
1338152 cache-misses # 0.101 % of all
cache refs
1326037487 cache-references
1.056355887 seconds time elapsed
1.014633000 seconds user
0.041805000 seconds sys
Performance counter stats for 'env
GLIBC_TUNABLES=glibc.malloc.thp_pagesize=1 ./testrun.sh ./hugepages
':
1016 faults
1746 dTLB-load-misses
399052 cache-misses # 0.030 % of all
cache refs
1316059877 cache-references
1.057810501 seconds time elapsed
1.012175000 seconds user
0.045624000 seconds sys
It is worth to note that the powerpc64 machine has 'always' set
on '/sys/kernel/mm/transparent_hugepage/enabled'.
Norbert Manthey's paper has more information with a more thoroughly
performance analysis.
This is a based of the previous RFC to enable Transparent Huge Page
with madvise [3], with fixes and improvements:
* Remove the usage of MAP_HUGE_SHIFT since kernel only uses it
when MAP_HUGETLB is also used.
* Moved the option to a tunable and add documentation.
* Remove the address alignmente before madvise call, since kernels
already does it.
* Added a arch-specific hook to return the THP huge page size.
* Avoid calling sbrk() twice to align to THP and remove a lot of
unecessary internal state.
* Add a NEWS entry.
[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/admin-guide/mm/transhuge.rst
[2] https://sourceware.org/pipermail/libc-alpha/2021-July/129041.html
[3] https://arxiv.org/pdf/2004.14378.pdf
[4] https://sourceware.org/pipermail/libc-alpha/2020-May/113539.html
Adhemerval Zanella (3):
malloc: Add madvise support for Transparent Huge Pages
malloc: Add THP/madvise support for sbrk
malloc: Add arch-specific malloc_verify_thp_pagesize for Linux
NEWS | 5 +-
elf/dl-tunables.list | 5 ++
elf/tst-rtld-list-tunables.exp | 1 +
malloc/arena.c | 5 ++
malloc/malloc-internal.h | 1 +
malloc/malloc.c | 85 ++++++++++++++++++--
manual/tunables.texi | 11 +++
sysdeps/generic/malloc-thp.h | 32 ++++++++
sysdeps/unix/sysv/linux/aarch64/malloc-thp.h | 40 +++++++++
sysdeps/unix/sysv/linux/mips/malloc-thp.h | 39 +++++++++
sysdeps/unix/sysv/linux/powerpc/malloc-thp.h | 56 +++++++++++++
sysdeps/unix/sysv/linux/riscv/malloc-thp.h | 32 ++++++++
sysdeps/unix/sysv/linux/s390/malloc-thp.h | 33 ++++++++
sysdeps/unix/sysv/linux/sparc/malloc-thp.h | 36 +++++++++
sysdeps/unix/sysv/linux/x86/malloc-thp.h | 32 ++++++++
15 files changed, 407 insertions(+), 6 deletions(-)
create mode 100644 sysdeps/generic/malloc-thp.h
create mode 100644 sysdeps/unix/sysv/linux/aarch64/malloc-thp.h
create mode 100644 sysdeps/unix/sysv/linux/mips/malloc-thp.h
create mode 100644 sysdeps/unix/sysv/linux/powerpc/malloc-thp.h
create mode 100644 sysdeps/unix/sysv/linux/riscv/malloc-thp.h
create mode 100644 sysdeps/unix/sysv/linux/s390/malloc-thp.h
create mode 100644 sysdeps/unix/sysv/linux/sparc/malloc-thp.h
create mode 100644 sysdeps/unix/sysv/linux/x86/malloc-thp.h
--
2.30.2
next reply other threads:[~2021-08-13 21:05 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-08-13 21:04 Adhemerval Zanella via Libc-alpha [this message]
2021-08-13 21:04 ` [PATCH 1/3] malloc: Add madvise support for Transparent Huge Pages Adhemerval Zanella via Libc-alpha
2021-08-13 21:04 ` [PATCH 2/3] malloc: Add THP/madvise support for sbrk Adhemerval Zanella via Libc-alpha
2021-08-13 21:04 ` [PATCH 3/3] malloc: Add arch-specific malloc_verify_thp_pagesize for Linux Adhemerval Zanella via Libc-alpha
2021-08-13 21:37 ` [PATCH 0/3] malloc: improve THP effectiveness Guillaume Morin
2021-08-16 20:55 ` Adhemerval Zanella via Libc-alpha
2021-08-17 4:00 ` Siddhesh Poyarekar via Libc-alpha
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://www.gnu.org/software/libc/involved.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210813210429.1147112-1-adhemerval.zanella@linaro.org \
--to=libc-alpha@sourceware.org \
--cc=adhemerval.zanella@linaro.org \
--cc=guillaume@morinfr.org \
--cc=nmanthey@conp-solutions.com \
--cc=siddhesh@sourceware.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).