Re: [PATCH] nptl: Disable THP on thread stack if it incurs in large RSS usage

[Adhemerval Zanella Netto via Libc-alpha <libc-alpha@sourceware.org>]

On 17/05/23 11:22, Wilco Dijkstra wrote:
> Hi Adhemerval,
> 
>> AFAIU the issue is after the stack is allocated with huge pages, the
>> kernel needs to fallback to standard pages because the guard 'page'
>> will be also within the same huge page allocated for the stack.  
> 
> The stack allocation explicitly never overlaps with the guard page, ie. there
> is no such fallback. All that matters is the mapped address range of the
> stack - if this fits huge pages, you'll get them.
> 
>> My understanding is, once kernel needs to fallback to use default pages, 
>> it allocates *all* the large page range.  This is what the RSS increase
>> make me believe, I am not sure if there is technical limitation to just 
>> making the range COW (since at the time of guard protection setup, no
>> the page has not been touched yet).
> 
> That's not what happens. The RSS size increases because you actually get
> a huge page (as requested). There is no fallback to standard pages.

But the threads themselves do not end up using all the VMA region allocated
for them.  Using the test program you can see it:

$ cat /proc/meminfo  | grep AnonHugePages
AnonHugePages:     43008 kB
$ ./tststackalloc &
[...]
[statm] RSS: 1049 pages (4296704 bytes = 4 MB)
[smaps] RSS: 5033984 bytes = 4 MB
[...]
$ cat /proc/meminfo  | grep AnonHugePages
AnonHugePages:     45056 kB

So even if the stack is not aligned to default large page, THP will still
back up the thread allocation.  The issues is, if the mmap is also aligned 
to THP size, the guard setup will trigger the issue that will increase RSS.
This seems to be same conclusion OpenJVM and some kernel discussion has
reached as well [1] [2].

> 
>>> So the real question is when do huge pages make sense for stacks?
>>
>> But that's not what the patch is trying to do, it only tries tot mitigate
>> a specific corner case where THP will be ineffective.  I agree with
> 
> So far there is no evidence this corner case exists, but even ignoring that,
> the expression used is incorrect.
> 
>> Cupertino that this question is really hard to answer and it will be
>> really depended of the workload and/or runtime characteristics that we will
>> need to plug in kernel feedback to have some answer.
> 
> It should be feasible to run benchmarks to get an idea whether huge stack pages
> help or not. And similarly whether the RSS increase is worth it or not.

Another option, hinted in both discussion and brought by Florian as well is to
add a pthread extension to force huge page disabl (something like pthread_attr_setflags
to make is extensible).

[1] https://bugs.openjdk.org/browse/JDK-8303215?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel&showAll=true
[2] https://lore.kernel.org/linux-mm/278ec047-4c5d-ab71-de36-094dbed4067c@redhat.com/T/