Re: [PATCH 0/3] malloc: improve THP effectiveness

[Adhemerval Zanella via Libc-alpha <libc-alpha@sourceware.org>]

On 13/08/2021 18:37, Guillaume Morin wrote:
> Hello Adhemerval,
> 
> On 13 Aug 18:04, Adhemerval Zanella wrote:
>> Although it does improve THP effectiveness, it does not provide the same
>> features from libhugetlsfs morecore implementation [2], since it does
>> use MAP_HUGETLB explicit on mmap.  And I think this is not what we want
>> for glibc, it requires additional setup from the admin to mount the
>> hugetlsfs and reserve the pages with it outside from glibc scope.
> 
> I certainly do appreciate the effort. But unfortunately this is not a
> usable replacement for most libhugetlblfs users (who actually want to use
> hugetlbfs).

Yes, that's why I explicit stated this is not a replacement.  But
I had the misconception that MAP_HUGETLB would require to to use
solely with mmap files opened on libhugetls filesystem and that's
why I wrote that I think it is not meant to glibc.

However reading the kernel documentation properly and after some
experiment, I think we add another tunable to use MAP_HUGETLB
as first allocation option.

> 
> First, I'll argue to have THP supported directly in the allocator is
> probably a nice-to-have feature for THP users but probably not that
> critical considering you can just madvise() the memory *after*
> it's been allocated. Alternatively any malloc interposition scheme can
> do this trivially: afaik there were never an actual *need* for a
> morecore implementation in this case.
> There is no such possibility with hugetlbfs. It's either mmap() with
> MAP_HUGETLB or not.

Yeah, I am aware. The idea is mainly to abstract to requirement to
query the kernel or handle the multiple pagesize from different
architectures and also possible handle the sbrk() calls for main
arena.  We can also add more tuning in the future if we find some
scenarios where THP need tuning.

> 
> Second, THP is not a drop-in replacement for hugetblfs. hugetlbfs has
> desirable properties that simply do not exist for THP. Just a few
> examples: 1) A hugetlbfs allocation gives you a huge page or not at
> allocation time but this is forever. There is no splitting, re-merging
> by the VM: no TLB shootdowns for running processes etc. 2) Fast
> allocation: there is a dedicated pool of these pages.  There is no
> competition with the rest of the processes unlike THP 3) No swapping all
> hugetlbfs pages.
> 
> I would really like to discuss and/or maybe implement some schemable
> that allows to optionally use MAP_HUGETLB for all allocations (which
> would be a definitive improvement over what libhugetlbfs was doing) if
> that's workable for you.

I am reworking this patchset and I intend to add an option to use
MAP_HUGETLB as well.