* Re: [PATCH 2/2] syscall: use pure Perl sendmsg/recvmsg on *BSD
2024-01-29 21:23 4% ` [PATCH 2/2] syscall: use " Eric Wong
@ 2024-04-06 0:43 7% ` Gaelan Steele
0 siblings, 0 replies; 3+ results
From: Gaelan Steele @ 2024-04-06 0:43 UTC (permalink / raw)
To: Eric Wong; +Cc: meta
> On Jan 29, 2024, at 9:23 PM, Eric Wong <e@80x24.org> wrote:
>
> While syscall symbols (e.g. SYS_*) have changed on us in FreeBSD
> during the history of Sys::Syscall and this project and did bite
> us in some cases; the actual numbers don't get recycled for new
> syscalls. We're also fortunate that sendmsg and recvmsg syscalls
> and associated msghdr and cmsg structs predate the BSD forks and
> are compatible across all the BSDs I've tried.
>
> OpenBSD routes Perl `syscall' through libc; while NetBSD + FreeBSD
> document procedures for maintaining backwards compatibility.
> It looks like Dragonfly follows FreeBSD, here.
>
> Tested on i386 OpenBSD, and amd64 {Free,Net,Open,Dragonfly}BSD
>
> This enables *BSD users to use lei, -cindex and future SCM_RIGHTS-only
> features without needing Inline::C.
Unfortunately this patch broke public-inbox on Darwin:
Bareword "SIZEOF_cmsghdr" not allowed while "strict subs" in use at /tmp/public-inbox/lib/PublicInbox/Syscall.pm line 456.
BEGIN not safe after errors--compilation aborted at /tmp/public-inbox/lib/PublicInbox/Syscall.pm line 460.
Compilation failed in require at /tmp/public-inbox/lib/PublicInbox/DS.pm line 31.
BEGIN failed--compilation aborted at /tmp/public-inbox/lib/PublicInbox/DS.pm line 32.
Compilation failed in require at /tmp/public-inbox/lib/PublicInbox/Daemon.pm line 17.
BEGIN failed--compilation aborted at /tmp/public-inbox/lib/PublicInbox/Daemon.pm line 17.
Compilation failed in require at /tmp/public-inbox/script/public-inbox-httpd line 7.
BEGIN failed--compilation aborted at /tmp/public-inbox/script/public-inbox-httpd line 7.
I’m not enough of a Perl person to fully untangle this. As
best I can tell, the intent is that non-Linux/BSD OSes should
still work with Inline::C, but this doesn’t work in practice
due to a bug?
It may also be possible to use the BSD approach on Darwin -
Darwin ascribes to the BSD school of thought where libc is the
only Officially Stable interface, but if you can get away with
it on the real BSDs maybe you can get away with it on fake BSD
too.
Best wishes,
Gaelan
^ permalink raw reply [relevance 7%]
* [PATCH 2/2] syscall: use pure Perl sendmsg/recvmsg on *BSD
2024-01-29 21:23 6% ` [PATCH 0/2] pure Perl sendmsg/recvmsg on *BSD Eric Wong
@ 2024-01-29 21:23 4% ` Eric Wong
2024-04-06 0:43 7% ` Gaelan Steele
0 siblings, 1 reply; 3+ results
From: Eric Wong @ 2024-01-29 21:23 UTC (permalink / raw)
To: meta
While syscall symbols (e.g. SYS_*) have changed on us in FreeBSD
during the history of Sys::Syscall and this project and did bite
us in some cases; the actual numbers don't get recycled for new
syscalls. We're also fortunate that sendmsg and recvmsg syscalls
and associated msghdr and cmsg structs predate the BSD forks and
are compatible across all the BSDs I've tried.
OpenBSD routes Perl `syscall' through libc; while NetBSD + FreeBSD
document procedures for maintaining backwards compatibility.
It looks like Dragonfly follows FreeBSD, here.
Tested on i386 OpenBSD, and amd64 {Free,Net,Open,Dragonfly}BSD
This enables *BSD users to use lei, -cindex and future SCM_RIGHTS-only
features without needing Inline::C.
[1] https://cvsweb.openbsd.org/src/gnu/usr.bin/perl/gen_syscall_emulator.pl
[2] https://www.netbsd.org/docs/internals/en/chap-processes.html#syscall_versioning
[3] https://wiki.freebsd.org/AddingSyscalls#Backward_compatibily
---
devel/sysdefs-list | 9 +++-
lib/PublicInbox/Syscall.pm | 102 +++++++++++++++++++++++--------------
t/cmd_ipc.t | 9 ++--
3 files changed, 74 insertions(+), 46 deletions(-)
diff --git a/devel/sysdefs-list b/devel/sysdefs-list
index 61532cf2..ba51de6c 100755
--- a/devel/sysdefs-list
+++ b/devel/sysdefs-list
@@ -2,8 +2,6 @@
# License: AGPL-3.0+ <http://www.gnu.org/licenses/agpl-3.0.txt>
# Dump system-specific constant numbers this is to maintain
# PublicInbox::Syscall and any other system-specific pieces.
-# DO NOT USE syscall numbers for *BSDs, none of the current BSD kernels
-# we know about promise stable syscall numbers (unlike Linux).
# However, sysconf(3) constants are stable ABI on all safe to dump.
eval 'exec perl -S $0 ${1+"$@"}' # no shebang
if 0; # running under some shell
@@ -179,5 +177,12 @@ int main(void)
PR_NUM(cmsg_type);
STRUCT_END;
+ {
+ struct cmsghdr cmsg;
+ uintptr_t cmsg_data_off;
+ cmsg_data_off = (uintptr_t)CMSG_DATA(&cmsg) - (uintptr_t)&cmsg;
+ D(cmsg_data_off);
+ }
+
return 0;
}
diff --git a/lib/PublicInbox/Syscall.pm b/lib/PublicInbox/Syscall.pm
index 9071e6b1..829cfa3c 100644
--- a/lib/PublicInbox/Syscall.pm
+++ b/lib/PublicInbox/Syscall.pm
@@ -22,7 +22,7 @@ use POSIX qw(ENOENT ENOSYS EINVAL O_NONBLOCK);
use Socket qw(SOL_SOCKET SCM_RIGHTS);
use Config;
our %SIGNUM = (WINCH => 28); # most Linux, {Free,Net,Open}BSD, *Darwin
-our $INOTIFY;
+our ($INOTIFY, %PACK);
# $VERSION = '0.25'; # Sys::Syscall version
our @EXPORT_OK = qw(epoll_ctl epoll_create epoll_wait
@@ -44,26 +44,21 @@ use constant {
EPOLL_CTL_MOD => 3,
SIZEOF_int => $Config{intsize},
SIZEOF_size_t => $Config{sizesize},
+ SIZEOF_ptr => $Config{ptrsize},
NUL => "\0",
};
-use constant {
- TMPL_size_t => SIZEOF_size_t == 8 ? 'Q' : 'L',
- BYTES_4_hole => SIZEOF_size_t == 8 ? 'L' : '',
- # cmsg_len, cmsg_level, cmsg_type
- SIZEOF_cmsghdr => SIZEOF_int * 2 + SIZEOF_size_t,
-};
-
-my @BYTES_4_hole = BYTES_4_hole ? (0) : ();
+use constant TMPL_size_t => SIZEOF_size_t == 8 ? 'Q' : 'L';
our ($SYS_epoll_create,
$SYS_epoll_ctl,
$SYS_epoll_wait,
$SYS_signalfd4,
$SYS_renameat2,
- $F_SETPIPE_SZ);
+ $F_SETPIPE_SZ,
+ $SYS_sendmsg,
+ $SYS_recvmsg);
-my ($SYS_sendmsg, $SYS_recvmsg);
my $SYS_fstatfs; # don't need fstatfs64, just statfs.f_type
my ($FS_IOC_GETFLAGS, $FS_IOC_SETFLAGS);
my $SFD_CLOEXEC = 02000000; # Perl does not expose O_CLOEXEC
@@ -78,7 +73,7 @@ if ($^O eq "linux") {
# boundaries.
my $u64_mod_8 = 0;
- if ($Config{ptrsize} == 4) {
+ if (SIZEOF_ptr == 4) {
# if we're running on an x86_64 kernel, but a 32-bit process,
# we need to use the x32 or i386 syscall numbers.
if ($machine eq 'x86_64') {
@@ -281,16 +276,52 @@ EOM
*epoll_wait = \&epoll_wait_mod4;
*epoll_ctl = \&epoll_ctl_mod4;
}
+} elsif ($^O =~ /\A(?:freebsd|openbsd|netbsd|dragonfly)\z/) {
+# don't use syscall.ph here, name => number mappings are not stable on *BSD
+# but the actual numbers are.
+# OpenBSD perl redirects syscall perlop to libc functions
+# https://cvsweb.openbsd.org/src/gnu/usr.bin/perl/gen_syscall_emulator.pl
+# https://www.netbsd.org/docs/internals/en/chap-processes.html#syscall_versioning
+# https://wiki.freebsd.org/AddingSyscalls#Backward_compatibily
+# (I'm assuming Dragonfly copies FreeBSD, here, too)
+ $SYS_recvmsg = 27;
+ $SYS_sendmsg = 28;
+}
+
+BEGIN {
+ if ($^O eq 'linux') {
+ %PACK = (
+ TMPL_cmsg_len => TMPL_size_t,
+ # cmsg_len, cmsg_level, cmsg_type
+ SIZEOF_cmsghdr => SIZEOF_int * 2 + SIZEOF_size_t,
+ CMSG_DATA_off => '',
+ TMPL_msghdr => 'PL' . # msg_name, msg_namelen
+ '@'.(2 * SIZEOF_ptr).'P'. # msg_iov
+ 'i'. # msg_iovlen
+ '@'.(4 * SIZEOF_ptr).'P'. # msg_control
+ 'L'. # msg_controllen (socklen_t)
+ 'i', # msg_flags
+ );
+ } elsif ($^O =~ /\A(?:freebsd|openbsd|netbsd|dragonfly)\z/) {
+ %PACK = (
+ TMPL_cmsg_len => 'L', # socklen_t
+ SIZEOF_cmsghdr => SIZEOF_int * 3,
+ CMSG_DATA_off => SIZEOF_ptr == 8 ? '@16' : '',
+ TMPL_msghdr => 'PL' . # msg_name, msg_namelen
+ '@'.(2 * SIZEOF_ptr).'P'. # msg_iov
+ TMPL_size_t. # msg_iovlen
+ '@'.(4 * SIZEOF_ptr).'P'. # msg_control
+ TMPL_size_t. # msg_controllen
+ 'i', # msg_flags
+
+ )
+ }
+ $PACK{CMSG_ALIGN_size} = SIZEOF_size_t;
}
# SFD_CLOEXEC is arch-dependent, so IN_CLOEXEC may be, too
$INOTIFY->{IN_CLOEXEC} //= 0x80000 if $INOTIFY;
-# use Inline::C for *BSD-only or general POSIX stuff.
-# Linux guarantees stable syscall numbering, BSDs only offer a stable libc
-# use devel/sysdefs-list on Linux to detect new syscall numbers and
-# other system constants
-
sub epoll_create {
syscall($SYS_epoll_create, $no_deprecated ? 0 : 100);
}
@@ -420,11 +451,13 @@ sub nodatacow_dir {
if (open my $fh, '<', $_[0]) { nodatacow_fh($fh) }
}
-sub CMSG_ALIGN ($) { ($_[0] + SIZEOF_size_t - 1) & ~(SIZEOF_size_t - 1) }
+use constant \%PACK;
+sub CMSG_ALIGN ($) { ($_[0] + CMSG_ALIGN_size - 1) & ~(CMSG_ALIGN_size - 1) }
use constant CMSG_ALIGN_SIZEOF_cmsghdr => CMSG_ALIGN(SIZEOF_cmsghdr);
sub CMSG_SPACE ($) { CMSG_ALIGN($_[0]) + CMSG_ALIGN_SIZEOF_cmsghdr }
sub CMSG_LEN ($) { CMSG_ALIGN_SIZEOF_cmsghdr + $_[0] }
-use constant msg_controllen => CMSG_SPACE(10 * SIZEOF_int) + 16; # 10 FDs
+use constant msg_controllen_max =>
+ CMSG_SPACE(10 * SIZEOF_int) + SIZEOF_cmsghdr; # space for 10 FDs
if (defined($SYS_sendmsg) && defined($SYS_recvmsg)) {
no warnings 'once';
@@ -436,20 +469,15 @@ require PublicInbox::CmdIPC4;
$_[2] // NUL, length($_[2] // NUL) || 1);
my $fd_space = scalar(@$fds) * SIZEOF_int;
my $msg_controllen = CMSG_SPACE($fd_space);
- my $cmsghdr = pack(TMPL_size_t . # cmsg_len
+ my $cmsghdr = pack(TMPL_cmsg_len .
'LL' . # cmsg_level, cmsg_type,
- ('i' x scalar(@$fds)) . # CMSG_DATA
+ CMSG_DATA_off.('i' x scalar(@$fds)). # CMSG_DATA
'@'.($msg_controllen - 1).'x1', # pad to space, not len
CMSG_LEN($fd_space), # cmsg_len
SOL_SOCKET, SCM_RIGHTS, # cmsg_{level,type}
@$fds); # CMSG_DATA
- my $mh = pack('PL' . # msg_name, msg_namelen (socklen_t (U32))
- BYTES_4_hole . # 4-byte padding on 64-bit
- 'P'.TMPL_size_t . # msg_iov, msg_iovlen,
- 'P'.TMPL_size_t . # msg_control, msg_controllen,
- 'i', # msg_flags
- NUL, 0, # msg_name, msg_namelen (unused)
- @BYTES_4_hole,
+ my $mh = pack(TMPL_msghdr,
+ undef, 0, # msg_name, msg_namelen (unused)
$iov, 1, # msg_iov, msg_iovlen
$cmsghdr, # msg_control
$msg_controllen,
@@ -465,18 +493,13 @@ require PublicInbox::CmdIPC4;
*recv_cmd4 = sub ($$$) {
my ($sock, undef, $len) = @_;
vec($_[1] //= '', $len - 1, 8) = 0;
- my $cmsghdr = "\0" x msg_controllen; # 10 * sizeof(int)
+ my $cmsghdr = "\0" x msg_controllen_max; # 10 * sizeof(int)
my $iov = pack('P'.TMPL_size_t, $_[1], $len);
- my $mh = pack('PL' . # msg_name, msg_namelen (socklen_t (U32))
- BYTES_4_hole . # 4-byte padding on 64-bit
- 'P'.TMPL_size_t . # msg_iov, msg_iovlen,
- 'P'.TMPL_size_t . # msg_control, msg_controllen,
- 'i', # msg_flags
- NUL, 0, # msg_name, msg_namelen (unused)
- @BYTES_4_hole,
+ my $mh = pack(TMPL_msghdr,
+ undef, 0, # msg_name, msg_namelen (unused)
$iov, 1, # msg_iov, msg_iovlen
$cmsghdr, # msg_control
- msg_controllen,
+ msg_controllen_max,
0); # msg_flags
my $r;
do {
@@ -489,8 +512,9 @@ require PublicInbox::CmdIPC4;
substr($_[1], $r, length($_[1]), '');
my @ret;
if ($r > 0) {
- my ($len, $lvl, $type, @fds) = unpack(TMPL_size_t . # cmsg_len
- 'LLi*', # cmsg_level, cmsg_type, @fds
+ my ($len, $lvl, $type, @fds) = unpack(TMPL_cmsg_len.
+ 'LL'. # cmsg_level, cmsg_type
+ CMSG_DATA_off.'i*', # @fds
$cmsghdr);
if ($lvl == SOL_SOCKET && $type == SCM_RIGHTS) {
$len -= CMSG_ALIGN_SIZEOF_cmsghdr;
diff --git a/t/cmd_ipc.t b/t/cmd_ipc.t
index 08a4dcc3..c973c6f0 100644
--- a/t/cmd_ipc.t
+++ b/t/cmd_ipc.t
@@ -143,14 +143,13 @@ SKIP: {
}
SKIP: {
- skip 'not Linux', 1 if $^O ne 'linux';
require_ok 'PublicInbox::Syscall';
$send = PublicInbox::Syscall->can('send_cmd4') or
- skip 'send_cmd4 not defined for arch', 1;
+ skip "send_cmd4 not defined for $^O arch", 1;
$recv = PublicInbox::Syscall->can('recv_cmd4') or
- skip 'recv_cmd4 not defined for arch', 1;
- $do_test->(SOCK_STREAM, 0, 'PP Linux stream');
- $do_test->(SOCK_SEQPACKET, 0, 'PP Linux seqpacket');
+ skip "recv_cmd4 not defined for $^O arch", 1;
+ $do_test->(SOCK_STREAM, 0, 'pure Perl stream');
+ $do_test->(SOCK_SEQPACKET, 0, 'pure Perl seqpacket');
}
done_testing;
^ permalink raw reply related [relevance 4%]
* [PATCH 0/2] pure Perl sendmsg/recvmsg on *BSD
@ 2024-01-29 21:23 6% ` Eric Wong
2024-01-29 21:23 4% ` [PATCH 2/2] syscall: use " Eric Wong
0 siblings, 1 reply; 3+ results
From: Eric Wong @ 2024-01-29 21:23 UTC (permalink / raw)
To: meta
I was wrong about the `syscall' Perl function being unusable on
*BSD. It turns out only the symbol names (e.g. from syscall.ph)
are unusable, but using the numbers is fine.
Eric Wong (2):
syscall: update formatting to match our codebase
syscall: use pure Perl sendmsg/recvmsg on *BSD
devel/sysdefs-list | 9 +-
lib/PublicInbox/Syscall.pm | 527 +++++++++++++++++++------------------
t/cmd_ipc.t | 9 +-
3 files changed, 286 insertions(+), 259 deletions(-)
^ permalink raw reply [relevance 6%]
Results 1-3 of 3 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2023-11-28 14:56 [PATCH 02/14] t/cindex*: require SCM_RIGHTS for these tests Eric Wong
2024-01-29 21:23 6% ` [PATCH 0/2] pure Perl sendmsg/recvmsg on *BSD Eric Wong
2024-01-29 21:23 4% ` [PATCH 2/2] syscall: use " Eric Wong
2024-04-06 0:43 7% ` Gaelan Steele
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).