Re: [PATCH] support for accessing CPU/core count (processor-count)

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 1100 bytes --]

On 10/10/21 5:21 AM, Stefan Kangas wrote:
> Omar Polo <op@omarpolo.com> writes:
>> OpenBSD disables hyperthreading by default so HW_NCPU is (almost) always
>> misleading.  For example, on my machine
>>
>>          % uname -a
>>          OpenBSD venera 7.0 GENERIC.MP#221 amd64
>>          % sysctl hw.ncpu
>>          hw.ncpu=8
>>          % sysctl hw.ncpuonline
>>          hw.ncpuonline=4
>>
>> and this has been the case for a while already (I mean, a couple of
>> years if not more.)
> 
> BTW, Gnulib doesn't seem to make this distinction (lib/nproc.c:313).
> Maybe that should be reported to the Gnulib developers?


No need for a bug report as I'm one of those developers. To fix the 
Gnulib nproc OpenBSD issue I installed the first attached patch into 
Gnulib, and to port Emacs's recently-added processor-count code to more 
platforms I installed the second attached patch into Emacs's emacs-28 
branch. Although I think the latter patch means that 
emacs/src/w32proc.c's w32-get-nproc function can be removed, I'll let 
the Emacs MS-Windows experts opine on that.

[-- Attachment #2: 0001-nproc-port-better-to-OpenBSD.patch --]
[-- Type: text/x-patch, Size: 1988 bytes --]

From 7fc3219bcc2bf448ef26cf30a2e5770fdda3f2b4 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Sun, 10 Oct 2021 10:43:47 -0700
Subject: [PATCH] nproc: port better to OpenBSD

Problem reported by Omar Polo in:
https://lists.gnu.org/r/emacs-devel/2021-10/msg00692.html
* lib/nproc.c (num_processors_ignoring_omp): Prefer HW_NCPUONLINE
to HW_NCPU, for OpenBSD.  Also, make mib const.
---
 ChangeLog   |  8 ++++++++
 lib/nproc.c | 19 +++++++++++++------
 2 files changed, 21 insertions(+), 6 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index 232d2dec3..da5b570ee 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,11 @@
+2021-10-10  Paul Eggert  <eggert@cs.ucla.edu>
+
+	nproc: port better to OpenBSD
+	Problem reported by Omar Polo in:
+	https://lists.gnu.org/r/emacs-devel/2021-10/msg00692.html
+	* lib/nproc.c (num_processors_ignoring_omp): Prefer HW_NCPUONLINE
+	to HW_NCPU, for OpenBSD.  Also, make mib const.
+
 2021-10-02  Paul Eggert  <eggert@cs.ucla.edu>
 
 	timer-time: port better to OpenBSD 6.9
diff --git a/lib/nproc.c b/lib/nproc.c
index e3ddb9288..a9e369dd3 100644
--- a/lib/nproc.c
+++ b/lib/nproc.c
@@ -310,12 +310,19 @@ num_processors_ignoring_omp (enum nproc_query query)
   { /* This works on Mac OS X, FreeBSD, NetBSD, OpenBSD.  */
     int nprocs;
     size_t len = sizeof (nprocs);
-    static int mib[2] = { CTL_HW, HW_NCPU };
-
-    if (sysctl (mib, ARRAY_SIZE (mib), &nprocs, &len, NULL, 0) == 0
-        && len == sizeof (nprocs)
-        && 0 < nprocs)
-      return nprocs;
+    static int const mib[][2] = {
+# ifdef HW_NCPUONLINE
+      { CTL_HW, HW_NCPUONLINE },
+# endif
+      { CTL_HW, HW_NCPU }
+    };
+    for (int i = 0; i < ARRAY_SIZE (mib); i++)
+      {
+        if (sysctl (mib[i], ARRAY_SIZE (mib[i]), &nprocs, &len, NULL, 0) == 0
+            && len == sizeof (nprocs)
+            && 0 < nprocs)
+          return nprocs;
+      }
   }
 #endif
 
-- 
2.30.2


[-- Attachment #3: 0001-New-function-num-processors.patch --]
[-- Type: text/x-patch, Size: 26157 bytes --]

From 96278de8ac2166c37925f2dfbc0eeb6d368142b9 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Sun, 10 Oct 2021 13:59:16 -0700
Subject: [PATCH] New function num-processors
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

This addresses a FIXME comment in lisp/emacs-lisp/comp.el,
relating to the number of subsidiary processes used by
comp-run-async-workers in native compilation.
* admin/merge-gnulib (GNULIB_MODULES): Add nproc.
* doc/lispref/processes.texi (Process Information), etc/NEWS:
Document num-processors.
* lib/gnulib.mk.in, m4/gnulib-comp.m4: Regenerate.
* lib/nproc.c, lib/nproc.h, m4/nproc.m4:
New files, copied from Gnulib by admin/merge-gnulib.
* lisp/emacs-lisp/comp.el (w32-get-nproc): Remove decl.
(comp-effective-async-max-jobs): Use num-processors.
* src/process.c: Include nproc.h.
(Fnum_processors): New function.
(syms_of_process): Define ‘all’, ‘current’, ‘num-processors’.
* src/w32proc.c (Fw32_get_nproc): Add FIXME comment.
* test/src/process-tests.el (process-num-processors): New test.
---
 admin/merge-gnulib         |   3 +-
 doc/lispref/processes.texi |  13 ++
 etc/NEWS                   |   4 +
 lib/gnulib.mk.in           |  11 +
 lib/nproc.c                | 403 +++++++++++++++++++++++++++++++++++++
 lib/nproc.h                |  46 +++++
 lisp/emacs-lisp/comp.el    |  15 +-
 m4/gnulib-comp.m4          |   5 +
 m4/nproc.m4                |  54 +++++
 src/process.c              |  18 ++
 src/w32proc.c              |   1 +
 test/src/process-tests.el  |   6 +
 12 files changed, 564 insertions(+), 15 deletions(-)
 create mode 100644 lib/nproc.c
 create mode 100644 lib/nproc.h
 create mode 100644 m4/nproc.m4

diff --git a/admin/merge-gnulib b/admin/merge-gnulib
index 886f37e28c..c9fe3b2f95 100755
--- a/admin/merge-gnulib
+++ b/admin/merge-gnulib
@@ -39,7 +39,8 @@ GNULIB_MODULES=
   free-posix fstatat fsusage fsync futimens
   getloadavg getopt-gnu getrandom gettime gettimeofday gitlog-to-changelog
   ieee754-h ignore-value intprops largefile libgmp lstat
-  manywarnings memmem-simple mempcpy memrchr minmax mkostemp mktime nstrftime
+  manywarnings memmem-simple mempcpy memrchr minmax mkostemp mktime
+  nproc nstrftime
   pathmax pipe2 pselect pthread_sigmask
   qcopy-acl readlink readlinkat regex
   sig2str sigdescr_np socklen stat-time std-gnu11 stdalign stddef stdio
diff --git a/doc/lispref/processes.texi b/doc/lispref/processes.texi
index 90c4215637..d90097d0b0 100644
--- a/doc/lispref/processes.texi
+++ b/doc/lispref/processes.texi
@@ -1047,6 +1047,19 @@ Process Information
 @end smallexample
 @end defun
 
+@defun num-processors &optional query
+This function returns the number of processors, a positive integer.
+Each usable thread execution unit counts as a processor.
+By default, the count includes the number of available processors,
+which you can override by setting the
+@url{https://www.openmp.org/spec-html/5.1/openmpse59.html,
+@env{OMP_NUM_THREADS} environment variable of OpenMP}.
+If the optional argument @var{query} is @code{current},
+this function ignores @env{OMP_NUM_THREADS};
+if @var{query} is @code{all}, this function also counts processors
+that are on the system but are not available to the current process.
+@end defun
+
 @defun get-process name
 This function returns the process named @var{name} (a string), or
 @code{nil} if there is none.  The argument @var{name} can also be a
diff --git a/etc/NEWS b/etc/NEWS
index 09537d7d31..791248f7dc 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -4094,6 +4094,10 @@ Parse a string as a mail address-like string.
 ** New function 'make-separator-line'.
 Make a string appropriate for usage as a visual separator line.
 
++++
+** New function 'num-processors'.
+Return the number of processors on the system.
+
 +++
 ** New function 'object-intervals'.
 This function returns a copy of the list of intervals (i.e., text
diff --git a/lib/gnulib.mk.in b/lib/gnulib.mk.in
index e9a1a5dc02..c7c7eb455b 100644
--- a/lib/gnulib.mk.in
+++ b/lib/gnulib.mk.in
@@ -129,6 +129,7 @@
 #  minmax \
 #  mkostemp \
 #  mktime \
+#  nproc \
 #  nstrftime \
 #  pathmax \
 #  pipe2 \
@@ -2378,6 +2379,16 @@ EXTRA_libgnu_a_SOURCES += mktime.c
 endif
 ## end   gnulib module mktime-internal
 
+## begin gnulib module nproc
+ifeq (,$(OMIT_GNULIB_MODULE_nproc))
+
+libgnu_a_SOURCES += nproc.c
+
+EXTRA_DIST += nproc.h
+
+endif
+## end   gnulib module nproc
+
 ## begin gnulib module nstrftime
 ifeq (,$(OMIT_GNULIB_MODULE_nstrftime))
 
diff --git a/lib/nproc.c b/lib/nproc.c
new file mode 100644
index 0000000000..a9e369dd3f
--- /dev/null
+++ b/lib/nproc.c
@@ -0,0 +1,403 @@
+/* Detect the number of processors.
+
+   Copyright (C) 2009-2021 Free Software Foundation, Inc.
+
+   This file is free software: you can redistribute it and/or modify
+   it under the terms of the GNU Lesser General Public License as
+   published by the Free Software Foundation; either version 2.1 of the
+   License, or (at your option) any later version.
+
+   This file is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+   GNU Lesser General Public License for more details.
+
+   You should have received a copy of the GNU Lesser General Public License
+   along with this program.  If not, see <https://www.gnu.org/licenses/>.  */
+
+/* Written by Glen Lenker and Bruno Haible.  */
+
+#include <config.h>
+#include "nproc.h"
+
+#include <limits.h>
+#include <stdlib.h>
+#include <unistd.h>
+
+#if HAVE_PTHREAD_GETAFFINITY_NP && 0
+# include <pthread.h>
+# include <sched.h>
+#endif
+#if HAVE_SCHED_GETAFFINITY_LIKE_GLIBC || HAVE_SCHED_GETAFFINITY_NP
+# include <sched.h>
+#endif
+
+#include <sys/types.h>
+
+#if HAVE_SYS_PSTAT_H
+# include <sys/pstat.h>
+#endif
+
+#if HAVE_SYS_SYSMP_H
+# include <sys/sysmp.h>
+#endif
+
+#if HAVE_SYS_PARAM_H
+# include <sys/param.h>
+#endif
+
+#if HAVE_SYS_SYSCTL_H && ! defined __GLIBC__
+# include <sys/sysctl.h>
+#endif
+
+#if defined _WIN32 && ! defined __CYGWIN__
+# define WIN32_LEAN_AND_MEAN
+# include <windows.h>
+#endif
+
+#include "c-ctype.h"
+
+#include "minmax.h"
+
+#define ARRAY_SIZE(a) (sizeof (a) / sizeof ((a)[0]))
+
+/* Return the number of processors available to the current process, based
+   on a modern system call that returns the "affinity" between the current
+   process and each CPU.  Return 0 if unknown or if such a system call does
+   not exist.  */
+static unsigned long
+num_processors_via_affinity_mask (void)
+{
+  /* glibc >= 2.3.3 with NPTL and NetBSD 5 have pthread_getaffinity_np,
+     but with different APIs.  Also it requires linking with -lpthread.
+     Therefore this code is not enabled.
+     glibc >= 2.3.4 has sched_getaffinity whereas NetBSD 5 has
+     sched_getaffinity_np.  */
+#if HAVE_PTHREAD_GETAFFINITY_NP && defined __GLIBC__ && 0
+  {
+    cpu_set_t set;
+
+    if (pthread_getaffinity_np (pthread_self (), sizeof (set), &set) == 0)
+      {
+        unsigned long count;
+
+# ifdef CPU_COUNT
+        /* glibc >= 2.6 has the CPU_COUNT macro.  */
+        count = CPU_COUNT (&set);
+# else
+        size_t i;
+
+        count = 0;
+        for (i = 0; i < CPU_SETSIZE; i++)
+          if (CPU_ISSET (i, &set))
+            count++;
+# endif
+        if (count > 0)
+          return count;
+      }
+  }
+#elif HAVE_PTHREAD_GETAFFINITY_NP && defined __NetBSD__ && 0
+  {
+    cpuset_t *set;
+
+    set = cpuset_create ();
+    if (set != NULL)
+      {
+        unsigned long count = 0;
+
+        if (pthread_getaffinity_np (pthread_self (), cpuset_size (set), set)
+            == 0)
+          {
+            cpuid_t i;
+
+            for (i = 0;; i++)
+              {
+                int ret = cpuset_isset (i, set);
+                if (ret < 0)
+                  break;
+                if (ret > 0)
+                  count++;
+              }
+          }
+        cpuset_destroy (set);
+        if (count > 0)
+          return count;
+      }
+  }
+#elif HAVE_SCHED_GETAFFINITY_LIKE_GLIBC /* glibc >= 2.3.4 */
+  {
+    cpu_set_t set;
+
+    if (sched_getaffinity (0, sizeof (set), &set) == 0)
+      {
+        unsigned long count;
+
+# ifdef CPU_COUNT
+        /* glibc >= 2.6 has the CPU_COUNT macro.  */
+        count = CPU_COUNT (&set);
+# else
+        size_t i;
+
+        count = 0;
+        for (i = 0; i < CPU_SETSIZE; i++)
+          if (CPU_ISSET (i, &set))
+            count++;
+# endif
+        if (count > 0)
+          return count;
+      }
+  }
+#elif HAVE_SCHED_GETAFFINITY_NP /* NetBSD >= 5 */
+  {
+    cpuset_t *set;
+
+    set = cpuset_create ();
+    if (set != NULL)
+      {
+        unsigned long count = 0;
+
+        if (sched_getaffinity_np (getpid (), cpuset_size (set), set) == 0)
+          {
+            cpuid_t i;
+
+            for (i = 0;; i++)
+              {
+                int ret = cpuset_isset (i, set);
+                if (ret < 0)
+                  break;
+                if (ret > 0)
+                  count++;
+              }
+          }
+        cpuset_destroy (set);
+        if (count > 0)
+          return count;
+      }
+  }
+#endif
+
+#if defined _WIN32 && ! defined __CYGWIN__
+  { /* This works on native Windows platforms.  */
+    DWORD_PTR process_mask;
+    DWORD_PTR system_mask;
+
+    if (GetProcessAffinityMask (GetCurrentProcess (),
+                                &process_mask, &system_mask))
+      {
+        DWORD_PTR mask = process_mask;
+        unsigned long count = 0;
+
+        for (; mask != 0; mask = mask >> 1)
+          if (mask & 1)
+            count++;
+        if (count > 0)
+          return count;
+      }
+  }
+#endif
+
+  return 0;
+}
+
+
+/* Return the total number of processors.  Here QUERY must be one of
+   NPROC_ALL, NPROC_CURRENT.  The result is guaranteed to be at least 1.  */
+static unsigned long int
+num_processors_ignoring_omp (enum nproc_query query)
+{
+  /* On systems with a modern affinity mask system call, we have
+         sysconf (_SC_NPROCESSORS_CONF)
+            >= sysconf (_SC_NPROCESSORS_ONLN)
+               >= num_processors_via_affinity_mask ()
+     The first number is the number of CPUs configured in the system.
+     The second number is the number of CPUs available to the scheduler.
+     The third number is the number of CPUs available to the current process.
+
+     Note! On Linux systems with glibc, the first and second number come from
+     the /sys and /proc file systems (see
+     glibc/sysdeps/unix/sysv/linux/getsysstats.c).
+     In some situations these file systems are not mounted, and the sysconf call
+     returns 1 or 2 (<https://sourceware.org/bugzilla/show_bug.cgi?id=21542>),
+     which does not reflect the reality.  */
+
+  if (query == NPROC_CURRENT)
+    {
+      /* Try the modern affinity mask system call.  */
+      {
+        unsigned long nprocs = num_processors_via_affinity_mask ();
+
+        if (nprocs > 0)
+          return nprocs;
+      }
+
+#if defined _SC_NPROCESSORS_ONLN
+      { /* This works on glibc, Mac OS X 10.5, FreeBSD, AIX, OSF/1, Solaris,
+           Cygwin, Haiku.  */
+        long int nprocs = sysconf (_SC_NPROCESSORS_ONLN);
+        if (nprocs > 0)
+          return nprocs;
+      }
+#endif
+    }
+  else /* query == NPROC_ALL */
+    {
+#if defined _SC_NPROCESSORS_CONF
+      { /* This works on glibc, Mac OS X 10.5, FreeBSD, AIX, OSF/1, Solaris,
+           Cygwin, Haiku.  */
+        long int nprocs = sysconf (_SC_NPROCESSORS_CONF);
+
+# if __GLIBC__ >= 2 && defined __linux__
+        /* On Linux systems with glibc, this information comes from the /sys and
+           /proc file systems (see glibc/sysdeps/unix/sysv/linux/getsysstats.c).
+           In some situations these file systems are not mounted, and the
+           sysconf call returns 1 or 2.  But we wish to guarantee that
+           num_processors (NPROC_ALL) >= num_processors (NPROC_CURRENT).  */
+        if (nprocs == 1 || nprocs == 2)
+          {
+            unsigned long nprocs_current = num_processors_via_affinity_mask ();
+
+            if (/* nprocs_current > 0 && */ nprocs_current > nprocs)
+              nprocs = nprocs_current;
+          }
+# endif
+
+        if (nprocs > 0)
+          return nprocs;
+      }
+#endif
+    }
+
+#if HAVE_PSTAT_GETDYNAMIC
+  { /* This works on HP-UX.  */
+    struct pst_dynamic psd;
+    if (pstat_getdynamic (&psd, sizeof psd, 1, 0) >= 0)
+      {
+        /* The field psd_proc_cnt contains the number of active processors.
+           In newer releases of HP-UX 11, the field psd_max_proc_cnt includes
+           deactivated processors.  */
+        if (query == NPROC_CURRENT)
+          {
+            if (psd.psd_proc_cnt > 0)
+              return psd.psd_proc_cnt;
+          }
+        else
+          {
+            if (psd.psd_max_proc_cnt > 0)
+              return psd.psd_max_proc_cnt;
+          }
+      }
+  }
+#endif
+
+#if HAVE_SYSMP && defined MP_NAPROCS && defined MP_NPROCS
+  { /* This works on IRIX.  */
+    /* MP_NPROCS yields the number of installed processors.
+       MP_NAPROCS yields the number of processors available to unprivileged
+       processes.  */
+    int nprocs =
+      sysmp (query == NPROC_CURRENT && getuid () != 0
+             ? MP_NAPROCS
+             : MP_NPROCS);
+    if (nprocs > 0)
+      return nprocs;
+  }
+#endif
+
+  /* Finally, as fallback, use the APIs that don't distinguish between
+     NPROC_CURRENT and NPROC_ALL.  */
+
+#if HAVE_SYSCTL && ! defined __GLIBC__ && defined HW_NCPU
+  { /* This works on Mac OS X, FreeBSD, NetBSD, OpenBSD.  */
+    int nprocs;
+    size_t len = sizeof (nprocs);
+    static int const mib[][2] = {
+# ifdef HW_NCPUONLINE
+      { CTL_HW, HW_NCPUONLINE },
+# endif
+      { CTL_HW, HW_NCPU }
+    };
+    for (int i = 0; i < ARRAY_SIZE (mib); i++)
+      {
+        if (sysctl (mib[i], ARRAY_SIZE (mib[i]), &nprocs, &len, NULL, 0) == 0
+            && len == sizeof (nprocs)
+            && 0 < nprocs)
+          return nprocs;
+      }
+  }
+#endif
+
+#if defined _WIN32 && ! defined __CYGWIN__
+  { /* This works on native Windows platforms.  */
+    SYSTEM_INFO system_info;
+    GetSystemInfo (&system_info);
+    if (0 < system_info.dwNumberOfProcessors)
+      return system_info.dwNumberOfProcessors;
+  }
+#endif
+
+  return 1;
+}
+
+/* Parse OMP environment variables without dependence on OMP.
+   Return 0 for invalid values.  */
+static unsigned long int
+parse_omp_threads (char const* threads)
+{
+  unsigned long int ret = 0;
+
+  if (threads == NULL)
+    return ret;
+
+  /* The OpenMP spec says that the value assigned to the environment variables
+     "may have leading and trailing white space".  */
+  while (*threads != '\0' && c_isspace (*threads))
+    threads++;
+
+  /* Convert it from positive decimal to 'unsigned long'.  */
+  if (c_isdigit (*threads))
+    {
+      char *endptr = NULL;
+      unsigned long int value = strtoul (threads, &endptr, 10);
+
+      if (endptr != NULL)
+        {
+          while (*endptr != '\0' && c_isspace (*endptr))
+            endptr++;
+          if (*endptr == '\0')
+            return value;
+          /* Also accept the first value in a nesting level,
+             since we can't determine the nesting level from env vars.  */
+          else if (*endptr == ',')
+            return value;
+        }
+    }
+
+  return ret;
+}
+
+unsigned long int
+num_processors (enum nproc_query query)
+{
+  unsigned long int omp_env_limit = ULONG_MAX;
+
+  if (query == NPROC_CURRENT_OVERRIDABLE)
+    {
+      unsigned long int omp_env_threads;
+      /* Honor the OpenMP environment variables, recognized also by all
+         programs that are based on OpenMP.  */
+      omp_env_threads = parse_omp_threads (getenv ("OMP_NUM_THREADS"));
+      omp_env_limit = parse_omp_threads (getenv ("OMP_THREAD_LIMIT"));
+      if (! omp_env_limit)
+        omp_env_limit = ULONG_MAX;
+
+      if (omp_env_threads)
+        return MIN (omp_env_threads, omp_env_limit);
+
+      query = NPROC_CURRENT;
+    }
+  /* Here query is one of NPROC_ALL, NPROC_CURRENT.  */
+  {
+    unsigned long nprocs = num_processors_ignoring_omp (query);
+    return MIN (nprocs, omp_env_limit);
+  }
+}
diff --git a/lib/nproc.h b/lib/nproc.h
new file mode 100644
index 0000000000..d7659a5cad
--- /dev/null
+++ b/lib/nproc.h
@@ -0,0 +1,46 @@
+/* Detect the number of processors.
+
+   Copyright (C) 2009-2021 Free Software Foundation, Inc.
+
+   This file is free software: you can redistribute it and/or modify
+   it under the terms of the GNU Lesser General Public License as
+   published by the Free Software Foundation; either version 2.1 of the
+   License, or (at your option) any later version.
+
+   This file is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+   GNU Lesser General Public License for more details.
+
+   You should have received a copy of the GNU Lesser General Public License
+   along with this program.  If not, see <https://www.gnu.org/licenses/>.  */
+
+/* Written by Glen Lenker and Bruno Haible.  */
+
+/* Allow the use in C++ code.  */
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/* A "processor" in this context means a thread execution unit, that is either
+   - an execution core in a (possibly multi-core) chip, in a (possibly multi-
+     chip) module, in a single computer, or
+   - a thread execution unit inside a core
+     (hyper-threading, see <https://en.wikipedia.org/wiki/Hyper-threading>).
+   Which of the two definitions is used, is unspecified.  */
+
+enum nproc_query
+{
+  NPROC_ALL,                 /* total number of processors */
+  NPROC_CURRENT,             /* processors available to the current process */
+  NPROC_CURRENT_OVERRIDABLE  /* likewise, but overridable through the
+                                OMP_NUM_THREADS environment variable */
+};
+
+/* Return the total number of processors.  The result is guaranteed to
+   be at least 1.  */
+extern unsigned long int num_processors (enum nproc_query query);
+
+#ifdef __cplusplus
+}
+#endif /* C++ */
diff --git a/lisp/emacs-lisp/comp.el b/lisp/emacs-lisp/comp.el
index 63d4a74b54..0052fd0f8d 100644
--- a/lisp/emacs-lisp/comp.el
+++ b/lisp/emacs-lisp/comp.el
@@ -3876,26 +3876,13 @@ comp-async-runnings
    do (remhash file-name comp-async-compilations))
   (hash-table-count comp-async-compilations))
 
-(declare-function w32-get-nproc "w32.c")
 (defvar comp-num-cpus nil)
 (defun comp-effective-async-max-jobs ()
   "Compute the effective number of async jobs."
   (if (zerop native-comp-async-jobs-number)
       (or comp-num-cpus
           (setf comp-num-cpus
-                ;; FIXME: we already have a function to determine
-                ;; the number of processors, see get_native_system_info in w32.c.
-                ;; The result needs to be exported to Lisp.
-                (max 1 (/ (cond ((eq 'windows-nt system-type)
-                                 (w32-get-nproc))
-                                ((executable-find "nproc")
-                                 (string-to-number
-                                  (shell-command-to-string "nproc")))
-                                ((eq 'berkeley-unix system-type)
-                                 (string-to-number
-                                  (shell-command-to-string "sysctl -n hw.ncpu")))
-                                (t 1))
-                          2))))
+		(max 1 (/ (num-processors) 2))))
     native-comp-async-jobs-number))
 
 (defvar comp-last-scanned-async-output nil)
diff --git a/m4/gnulib-comp.m4 b/m4/gnulib-comp.m4
index a795fe7651..e314edcfb5 100644
--- a/m4/gnulib-comp.m4
+++ b/m4/gnulib-comp.m4
@@ -139,6 +139,7 @@ AC_DEFUN
   # Code from module mktime-internal:
   # Code from module multiarch:
   # Code from module nocrash:
+  # Code from module nproc:
   # Code from module nstrftime:
   # Code from module open:
   # Code from module openat-h:
@@ -413,6 +414,7 @@ AC_DEFUN
   fi
   gl_TIME_MODULE_INDICATOR([mktime])
   gl_MULTIARCH
+  gl_NPROC
   gl_FUNC_GNU_STRFTIME
   gl_PATHMAX
   gl_FUNC_PIPE2
@@ -1221,6 +1223,8 @@ AC_DEFUN
   lib/mkostemp.c
   lib/mktime-internal.h
   lib/mktime.c
+  lib/nproc.c
+  lib/nproc.h
   lib/nstrftime.c
   lib/open.c
   lib/openat-priv.h
@@ -1370,6 +1374,7 @@ AC_DEFUN
   m4/mode_t.m4
   m4/multiarch.m4
   m4/nocrash.m4
+  m4/nproc.m4
   m4/nstrftime.m4
   m4/off_t.m4
   m4/open-cloexec.m4
diff --git a/m4/nproc.m4 b/m4/nproc.m4
new file mode 100644
index 0000000000..887c66bee8
--- /dev/null
+++ b/m4/nproc.m4
@@ -0,0 +1,54 @@
+# nproc.m4 serial 5
+dnl Copyright (C) 2009-2021 Free Software Foundation, Inc.
+dnl This file is free software; the Free Software Foundation
+dnl gives unlimited permission to copy and/or distribute it,
+dnl with or without modifications, as long as this notice is preserved.
+
+AC_DEFUN([gl_NPROC],
+[
+  gl_PREREQ_NPROC
+])
+
+# Prerequisites of lib/nproc.c.
+AC_DEFUN([gl_PREREQ_NPROC],
+[
+  dnl Persuade glibc <sched.h> to declare CPU_SETSIZE, CPU_ISSET etc.
+  AC_REQUIRE([AC_USE_SYSTEM_EXTENSIONS])
+
+  AC_CHECK_HEADERS([sys/pstat.h sys/sysmp.h sys/param.h],,,
+    [AC_INCLUDES_DEFAULT])
+  dnl <sys/sysctl.h> requires <sys/param.h> on OpenBSD 4.0.
+  AC_CHECK_HEADERS([sys/sysctl.h],,,
+    [AC_INCLUDES_DEFAULT
+     #if HAVE_SYS_PARAM_H
+     # include <sys/param.h>
+     #endif
+    ])
+
+  AC_CHECK_FUNCS([sched_getaffinity sched_getaffinity_np \
+                  pstat_getdynamic sysmp sysctl])
+
+  dnl Test whether sched_getaffinity has the expected declaration.
+  dnl glibc 2.3.[0-2]:
+  dnl   int sched_getaffinity (pid_t, unsigned int, unsigned long int *);
+  dnl glibc 2.3.3:
+  dnl   int sched_getaffinity (pid_t, cpu_set_t *);
+  dnl glibc >= 2.3.4:
+  dnl   int sched_getaffinity (pid_t, size_t, cpu_set_t *);
+  if test $ac_cv_func_sched_getaffinity = yes; then
+    AC_CACHE_CHECK([for glibc compatible sched_getaffinity],
+      [gl_cv_func_sched_getaffinity3],
+      [AC_COMPILE_IFELSE(
+         [AC_LANG_PROGRAM(
+            [[#include <errno.h>
+              #include <sched.h>]],
+            [[sched_getaffinity (0, 0, (cpu_set_t *) 0);]])],
+         [gl_cv_func_sched_getaffinity3=yes],
+         [gl_cv_func_sched_getaffinity3=no])
+      ])
+    if test $gl_cv_func_sched_getaffinity3 = yes; then
+      AC_DEFINE([HAVE_SCHED_GETAFFINITY_LIKE_GLIBC], [1],
+        [Define to 1 if sched_getaffinity has a glibc compatible declaration.])
+    fi
+  fi
+])
diff --git a/src/process.c b/src/process.c
index 221d4c7f6c..746cdc0428 100644
--- a/src/process.c
+++ b/src/process.c
@@ -90,6 +90,7 @@ #define HAVE_LOCAL_SOCKETS
 
 #include <c-ctype.h>
 #include <flexmember.h>
+#include <nproc.h>
 #include <sig2str.h>
 #include <verify.h>
 
@@ -8212,6 +8213,20 @@ DEFUN ("process-attributes", Fprocess_attributes,
   return system_process_attributes (pid);
 }
 
+DEFUN ("num-processors", Fnum_processors, Snum_processors, 0, 1, 0,
+       doc: /* Return the number of processors, a positive integer.
+Each usable thread execution unit counts as a processor.
+By default, count the number of available processors,
+overridable via the OMP_NUM_THREADS environment variable.
+If optional argument QUERY is `current', ignore OMP_NUM_THREADS.
+If QUERY is `all', also count processors not available.  */)
+  (Lisp_Object query)
+{
+  return make_uint (num_processors (EQ (query, Qall) ? NPROC_ALL
+				    : EQ (query, Qcurrent) ? NPROC_CURRENT
+				    : NPROC_CURRENT_OVERRIDABLE));
+}
+
 #ifdef subprocesses
 /* Arrange to catch SIGCHLD if this hasn't already been arranged.
    Invoke this after init_process_emacs, and after glib and/or GNUstep
@@ -8472,6 +8487,8 @@ syms_of_process (void)
   DEFSYM (Qpcpu, "pcpu");
   DEFSYM (Qpmem, "pmem");
   DEFSYM (Qargs, "args");
+  DEFSYM (Qall, "all");
+  DEFSYM (Qcurrent, "current");
 
   DEFVAR_BOOL ("delete-exited-processes", delete_exited_processes,
 	       doc: /* Non-nil means delete processes immediately when they exit.
@@ -8633,4 +8650,5 @@ #define ADD_SUBFEATURE(key, val) \
   defsubr (&Sprocess_inherit_coding_system_flag);
   defsubr (&Slist_system_processes);
   defsubr (&Sprocess_attributes);
+  defsubr (&Snum_processors);
 }
diff --git a/src/w32proc.c b/src/w32proc.c
index 702ea122e6..3b7d92a2aa 100644
--- a/src/w32proc.c
+++ b/src/w32proc.c
@@ -3878,6 +3878,7 @@ w32_compare_strings (const char *s1, const char *s2, char *locname,
   return val - 2;
 }
 
+/* FIXME: Remove, merging any of its special features into num-processors.  */
 DEFUN ("w32-get-nproc", Fw32_get_nproc,
        Sw32_get_nproc, 0, 0, 0,
        doc: /* Return the number of system's processor execution units.  */)
diff --git a/test/src/process-tests.el b/test/src/process-tests.el
index e39f57d23b..44f3ea2fbb 100644
--- a/test/src/process-tests.el
+++ b/test/src/process-tests.el
@@ -946,5 +946,11 @@ process-async-https-with-delay
       (when buf
         (kill-buffer buf)))))
 
+(ert-deftest process-num-processors ()
+  "Sanity checks for num-processors."
+  (should (equal (num-processors) (num-processors)))
+  (should (integerp (num-processors)))
+  (should (< 0 (num-processors))))
+
 (provide 'process-tests)
 ;;; process-tests.el ends here
-- 
2.30.2

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Omar Polo]

Paul Eggert <eggert@cs.ucla.edu> writes:

> On 10/10/21 5:21 AM, Stefan Kangas wrote:
>> Omar Polo <op@omarpolo.com> writes:
>>> OpenBSD disables hyperthreading by default so HW_NCPU is (almost) always
>>> misleading.  For example, on my machine
>>>
>>>          % uname -a
>>>          OpenBSD venera 7.0 GENERIC.MP#221 amd64
>>>          % sysctl hw.ncpu
>>>          hw.ncpu=8
>>>          % sysctl hw.ncpuonline
>>>          hw.ncpuonline=4
>>>
>>> and this has been the case for a while already (I mean, a couple of
>>> years if not more.)
>> BTW, Gnulib doesn't seem to make this distinction (lib/nproc.c:313).
>> Maybe that should be reported to the Gnulib developers?
>
>
> No need for a bug report as I'm one of those developers. To fix the
> Gnulib nproc OpenBSD issue I installed the first attached patch into
> Gnulib, and to port Emacs's recently-added processor-count code to
> more platforms I installed the second attached patch into Emacs's
> emacs-28 branch. Although I think the latter patch means that
> emacs/src/w32proc.c's w32-get-nproc function can be removed, I'll let
> the Emacs MS-Windows experts opine on that.

I had just cloned glib when I saw your mail :)

FWIW the first patch reads fine to me.  Thanks!

Re: [PATCH] support for accessing CPU/core count (processor-count)

[Arthur Miller]

Paul Eggert <eggert@cs.ucla.edu> writes:

> On 10/10/21 5:21 AM, Stefan Kangas wrote:
>> Omar Polo <op@omarpolo.com> writes:
>>> OpenBSD disables hyperthreading by default so HW_NCPU is (almost) always
>>> misleading.  For example, on my machine
>>>
>>>          % uname -a
>>>          OpenBSD venera 7.0 GENERIC.MP#221 amd64
>>>          % sysctl hw.ncpu
>>>          hw.ncpu=8
>>>          % sysctl hw.ncpuonline
>>>          hw.ncpuonline=4
>>>
>>> and this has been the case for a while already (I mean, a couple of
>>> years if not more.)
>> BTW, Gnulib doesn't seem to make this distinction (lib/nproc.c:313).
>> Maybe that should be reported to the Gnulib developers?
>
>
> No need for a bug report as I'm one of those developers. To fix the Gnulib nproc
> OpenBSD issue I installed the first attached patch into Gnulib, and to port
> Emacs's recently-added processor-count code to more platforms I installed the
> second attached patch into Emacs's emacs-28 branch. Although I think the latter
> patch means that emacs/src/w32proc.c's w32-get-nproc function can be removed,
> I'll let the Emacs MS-Windows experts opine on that.

So you just included entire nproc into Emacs :).  Ok. Anyway; it only gives
number of "logical cores" when "hyperthreading" is on not physical. I don't
think nproc can give you that either.