unofficial mirror of libc-alpha@sourceware.org
 help / color / mirror / Atom feed
* glibc at the Toolchains microconference at LPC 2019
@ 2019-06-26 16:01 Florian Weimer
  2019-06-26 16:39 ` Dmitry V. Levin
  2019-06-27 17:45 ` glibc at the Toolchains microconference at LPC 2019 Maciej Rozycki
  0 siblings, 2 replies; 38+ messages in thread
From: Florian Weimer @ 2019-06-26 16:01 UTC (permalink / raw)
  To: libc-alpha

glibc system call wrappers are on the agenda:

<https://www.linuxplumbersconf.org/blog/2019/toolchains-microconference-accepted-into-2019-linux-plumbers-conference/>

Will anyone from the glibc community attend and can set the right
expectations?

I won't be there because I'm traveling to Cauldron instead, which is
immediately after LPC, but in a different corner of the planet.

Thanks,
Florian

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: glibc at the Toolchains microconference at LPC 2019
  2019-06-26 16:01 glibc at the Toolchains microconference at LPC 2019 Florian Weimer
@ 2019-06-26 16:39 ` Dmitry V. Levin
  2019-06-26 16:50   ` Zack Weinberg
  2019-06-27 17:45 ` glibc at the Toolchains microconference at LPC 2019 Maciej Rozycki
  1 sibling, 1 reply; 38+ messages in thread
From: Dmitry V. Levin @ 2019-06-26 16:39 UTC (permalink / raw)
  To: libc-alpha

[-- Attachment #1: Type: text/plain, Size: 380 bytes --]

On Wed, Jun 26, 2019 at 06:01:28PM +0200, Florian Weimer wrote:
> glibc system call wrappers are on the agenda:
> 
> <https://www.linuxplumbersconf.org/blog/2019/toolchains-microconference-accepted-into-2019-linux-plumbers-conference/>
> 
> Will anyone from the glibc community attend and can set the right
> expectations?

What are the right expectations?


-- 
ldv

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: glibc at the Toolchains microconference at LPC 2019
  2019-06-26 16:39 ` Dmitry V. Levin
@ 2019-06-26 16:50   ` Zack Weinberg
  2019-06-26 16:56     ` Florian Weimer
                       ` (2 more replies)
  0 siblings, 3 replies; 38+ messages in thread
From: Zack Weinberg @ 2019-06-26 16:50 UTC (permalink / raw)
  To: Dmitry V. Levin; +Cc: GNU C Library

On Wed, Jun 26, 2019 at 12:39 PM Dmitry V. Levin <ldv@altlinux.org> wrote:
> On Wed, Jun 26, 2019 at 06:01:28PM +0200, Florian Weimer wrote:
> > glibc system call wrappers are on the agenda:
> >
> > <https://www.linuxplumbersconf.org/blog/2019/toolchains-microconference-accepted-into-2019-linux-plumbers-conference/>
> >
> > Will anyone from the glibc community attend and can set the right
> > expectations?
>
> What are the right expectations?

Well, _I_ think glibc should provide wrappers for all Linux system
calls, except those that cannot be used without stomping on internal
glibc data structures (e.g. set_tid_address, set_robust_list, brk) and
those that have been completely superseded by newer syscalls.  Other
people have disagreed with me pretty strenuously, but they haven't
done any of the work required to make forward progress on their more
conservative policies.  I am tempted to post a patch early in the 2.31
cycle that adds wrappers for everything, and then threaten to apply it
unilaterally unless I hear concrete objections within a week or so.

zw

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: glibc at the Toolchains microconference at LPC 2019
  2019-06-26 16:50   ` Zack Weinberg
@ 2019-06-26 16:56     ` Florian Weimer
  2019-06-26 17:40       ` Zack Weinberg
  2019-06-26 20:33       ` Carlos O'Donell
  2019-06-26 17:28     ` Yann Droneaud
  2019-06-26 17:41     ` Christian Brauner
  2 siblings, 2 replies; 38+ messages in thread
From: Florian Weimer @ 2019-06-26 16:56 UTC (permalink / raw)
  To: Zack Weinberg; +Cc: Dmitry V. Levin, GNU C Library

* Zack Weinberg:

> On Wed, Jun 26, 2019 at 12:39 PM Dmitry V. Levin <ldv@altlinux.org> wrote:
>> On Wed, Jun 26, 2019 at 06:01:28PM +0200, Florian Weimer wrote:
>> > glibc system call wrappers are on the agenda:
>> >
>> > <https://www.linuxplumbersconf.org/blog/2019/toolchains-microconference-accepted-into-2019-linux-plumbers-conference/>
>> >
>> > Will anyone from the glibc community attend and can set the right
>> > expectations?
>>
>> What are the right expectations?
>
> Well, _I_ think glibc should provide wrappers for all Linux system
> calls, except those that cannot be used without stomping on internal
> glibc data structures (e.g. set_tid_address, set_robust_list, brk) and
> those that have been completely superseded by newer syscalls.  Other
> people have disagreed with me pretty strenuously, but they haven't
> done any of the work required to make forward progress on their more
> conservative policies.  I am tempted to post a patch early in the 2.31
> cycle that adds wrappers for everything, and then threaten to apply it
> unilaterally unless I hear concrete objections within a week or so.

In my experience, it's been difficult to get reviewers.  So what the
project says it wants and what the project actually makes happen is
rather different.

There is currently a requirement that every wrapper needs a manual entry
(and, presumably, a test case, although I have not tested the waters on
that).  membarrier is not included only because we could not agree on
the manual text.

Thanks,
Florian

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: glibc at the Toolchains microconference at LPC 2019
  2019-06-26 16:50   ` Zack Weinberg
  2019-06-26 16:56     ` Florian Weimer
@ 2019-06-26 17:28     ` Yann Droneaud
  2019-06-26 17:41     ` Christian Brauner
  2 siblings, 0 replies; 38+ messages in thread
From: Yann Droneaud @ 2019-06-26 17:28 UTC (permalink / raw)
  To: Zack Weinberg; +Cc: GNU C Library, Dmitry V. Levin

Hi,

Le mercredi 26 juin 2019 à 12:50 -0400, Zack Weinberg a écrit :
> On Wed, Jun 26, 2019 at 12:39 PM Dmitry V. Levin <ldv@altlinux.org> wrote:
> > On Wed, Jun 26, 2019 at 06:01:28PM +0200, Florian Weimer wrote:
> > > glibc system call wrappers are on the agenda:
> > > 
> > > <https://www.linuxplumbersconf.org/blog/2019/toolchains-microconference-accepted-into-2019-linux-plumbers-conference/>
> > > 
> > > Will anyone from the glibc community attend and can set the right
> > > expectations?
> > 
> > What are the right expectations?
> 
> Well, _I_ think glibc should provide wrappers for all Linux system
> calls, except those that cannot be used without stomping on internal
> glibc data structures (e.g. set_tid_address, set_robust_list, brk) and
> those that have been completely superseded by newer syscalls.  

I believe many more syscalls must not be exposed by glibc by being too
specific, architecture dedicated syscalls in particular.

I strongly think that syscall wrappers that cannot be implemented with
a single line of code should be avoided, as it's a sign a dedicated
library would probably be better.

Bigger wrappers should only be allowed when the provided syscall is
intended to emulate some standard and/or cross-platform API.

So, yes, you could probably say I'm on "conservative" side :)

Regards.

-- 
Yann Droneaud
OPTEYA



^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: glibc at the Toolchains microconference at LPC 2019
  2019-06-26 16:56     ` Florian Weimer
@ 2019-06-26 17:40       ` Zack Weinberg
  2019-06-26 20:33       ` Carlos O'Donell
  1 sibling, 0 replies; 38+ messages in thread
From: Zack Weinberg @ 2019-06-26 17:40 UTC (permalink / raw)
  To: Florian Weimer; +Cc: GNU C Library

On Wed, Jun 26, 2019 at 12:56 PM Florian Weimer <fweimer@redhat.com> wrote:
> * Zack Weinberg:
> > Well, _I_ think glibc should provide wrappers for all Linux system
> > calls, except those that cannot be used without stomping on internal
> > glibc data structures (e.g. set_tid_address, set_robust_list, brk) and
> > those that have been completely superseded by newer syscalls.  Other
> > people have disagreed with me pretty strenuously, but they haven't
> > done any of the work required to make forward progress on their more
> > conservative policies.  I am tempted to post a patch early in the 2.31
> > cycle that adds wrappers for everything, and then threaten to apply it
> > unilaterally unless I hear concrete objections within a week or so.
>
> In my experience, it's been difficult to get reviewers.  So what the
> project says it wants and what the project actually makes happen is
> rather different.

For the record, the shade in my message was not aimed at you.  I think
you're one of the only people who's managed to make forward progress
on adding syscall wrappers in the past several years.

zw

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: glibc at the Toolchains microconference at LPC 2019
  2019-06-26 16:50   ` Zack Weinberg
  2019-06-26 16:56     ` Florian Weimer
  2019-06-26 17:28     ` Yann Droneaud
@ 2019-06-26 17:41     ` Christian Brauner
  2019-06-26 21:04       ` Carlos O'Donell
  2 siblings, 1 reply; 38+ messages in thread
From: Christian Brauner @ 2019-06-26 17:41 UTC (permalink / raw)
  To: libc-alpha, Zack Weinberg, Dmitry V. Levin; +Cc: GNU C Library

On June 26, 2019 6:50:23 PM GMT+02:00, Zack Weinberg <zackw@panix.com> wrote:
>On Wed, Jun 26, 2019 at 12:39 PM Dmitry V. Levin <ldv@altlinux.org>
>wrote:
>> On Wed, Jun 26, 2019 at 06:01:28PM +0200, Florian Weimer wrote:
>> > glibc system call wrappers are on the agenda:
>> >
>> >
><https://www.linuxplumbersconf.org/blog/2019/toolchains-microconference-accepted-into-2019-linux-plumbers-conference/>
>> >
>> > Will anyone from the glibc community attend and can set the right
>> > expectations?
>>
>> What are the right expectations?
>
>Well, _I_ think glibc should provide wrappers for all Linux system
>calls, except those that cannot be used without stomping on internal
>glibc data structures (e.g. set_tid_address, set_robust_list, brk) and
>those that have been completely superseded by newer syscalls.  Other
>people have disagreed with me pretty strenuously, but they haven't
>done any of the work required to make forward progress on their more
>conservative policies.  I am tempted to post a patch early in the 2.31
>cycle that adds wrappers for everything, and then threaten to apply it
>unilaterally unless I hear concrete objections within a week or so.
>
>zw

Strongly agree.

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: glibc at the Toolchains microconference at LPC 2019
  2019-06-26 16:56     ` Florian Weimer
  2019-06-26 17:40       ` Zack Weinberg
@ 2019-06-26 20:33       ` Carlos O'Donell
  2019-06-26 20:39         ` Christian Brauner
  1 sibling, 1 reply; 38+ messages in thread
From: Carlos O'Donell @ 2019-06-26 20:33 UTC (permalink / raw)
  To: Florian Weimer, Zack Weinberg; +Cc: Dmitry V. Levin, GNU C Library

On 6/26/19 12:56 PM, Florian Weimer wrote:
> * Zack Weinberg:
> 
>> On Wed, Jun 26, 2019 at 12:39 PM Dmitry V. Levin <ldv@altlinux.org> wrote:
>>> On Wed, Jun 26, 2019 at 06:01:28PM +0200, Florian Weimer wrote:
>>>> glibc system call wrappers are on the agenda:
>>>>
>>>> <https://www.linuxplumbersconf.org/blog/2019/toolchains-microconference-accepted-into-2019-linux-plumbers-conference/>
>>>>
>>>> Will anyone from the glibc community attend and can set the right
>>>> expectations?
>>>
>>> What are the right expectations?
>>
>> Well, _I_ think glibc should provide wrappers for all Linux system
>> calls, except those that cannot be used without stomping on internal
>> glibc data structures (e.g. set_tid_address, set_robust_list, brk) and
>> those that have been completely superseded by newer syscalls.  Other
>> people have disagreed with me pretty strenuously, but they haven't
>> done any of the work required to make forward progress on their more
>> conservative policies.  I am tempted to post a patch early in the 2.31
>> cycle that adds wrappers for everything, and then threaten to apply it
>> unilaterally unless I hear concrete objections within a week or so.
> 
> In my experience, it's been difficult to get reviewers.  So what the
> project says it wants and what the project actually makes happen is
> rather different.

It is difficult to get reviewers for *all* patches.

Therefore I don't think this is particular to syscall wrappers.

I've tried hard to review many of your syscall wrappers and make good
on the promise we gave to the kernel community that we would do so.

Lastly, if you do reviews please provide your "Reviewed-by" markers
since it will let me run metrics on how many people we have reviewing
and who they are, and use that information to for a long-term strategy
for getting more reviewers.

> There is currently a requirement that every wrapper needs a manual entry
> (and, presumably, a test case, although I have not tested the waters on
> that).  membarrier is not included only because we could not agree on
> the manual text.

And rightly so. I would hope that we all agree that we need documentation
and testing of interfaces in order to provide our users with the information
they need to use these interfaces.

-- 
Cheers,
Carlos.

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: glibc at the Toolchains microconference at LPC 2019
  2019-06-26 20:33       ` Carlos O'Donell
@ 2019-06-26 20:39         ` Christian Brauner
  2019-06-26 20:58           ` Carlos O'Donell
  0 siblings, 1 reply; 38+ messages in thread
From: Christian Brauner @ 2019-06-26 20:39 UTC (permalink / raw)
  To: libc-alpha, Carlos O'Donell, Florian Weimer, Zack Weinberg
  Cc: Dmitry V. Levin, GNU C Library

On June 26, 2019 10:33:37 PM GMT+02:00, Carlos O'Donell <carlos@redhat.com> wrote:
>On 6/26/19 12:56 PM, Florian Weimer wrote:
>> * Zack Weinberg:
>> 
>>> On Wed, Jun 26, 2019 at 12:39 PM Dmitry V. Levin <ldv@altlinux.org>
>wrote:
>>>> On Wed, Jun 26, 2019 at 06:01:28PM +0200, Florian Weimer wrote:
>>>>> glibc system call wrappers are on the agenda:
>>>>>
>>>>>
><https://www.linuxplumbersconf.org/blog/2019/toolchains-microconference-accepted-into-2019-linux-plumbers-conference/>
>>>>>
>>>>> Will anyone from the glibc community attend and can set the right
>>>>> expectations?
>>>>
>>>> What are the right expectations?
>>>
>>> Well, _I_ think glibc should provide wrappers for all Linux system
>>> calls, except those that cannot be used without stomping on internal
>>> glibc data structures (e.g. set_tid_address, set_robust_list, brk)
>and
>>> those that have been completely superseded by newer syscalls.  Other
>>> people have disagreed with me pretty strenuously, but they haven't
>>> done any of the work required to make forward progress on their more
>>> conservative policies.  I am tempted to post a patch early in the
>2.31
>>> cycle that adds wrappers for everything, and then threaten to apply
>it
>>> unilaterally unless I hear concrete objections within a week or so.
>> 
>> In my experience, it's been difficult to get reviewers.  So what the
>> project says it wants and what the project actually makes happen is
>> rather different.
>
>It is difficult to get reviewers for *all* patches.
>
>Therefore I don't think this is particular to syscall wrappers.
>
>I've tried hard to review many of your syscall wrappers and make good
>on the promise we gave to the kernel community that we would do so.
>
>Lastly, if you do reviews please provide your "Reviewed-by" markers
>since it will let me run metrics on how many people we have reviewing
>and who they are, and use that information to for a long-term strategy
>for getting more reviewers.
>
>> There is currently a requirement that every wrapper needs a manual
>entry
>> (and, presumably, a test case, although I have not tested the waters
>on
>> that).  membarrier is not included only because we could not agree on
>> the manual text.
>
>And rightly so. I would hope that we all agree that we need
>documentation
>and testing of interfaces in order to provide our users with the
>information
>they need to use these interfaces.

Difficult to get reviewers in the sense of kernel people who wrote the syscalls?
I'm trying as hard as I can to bridge the libc/kernel barrier
by always cc'ing e.g. Florian or Dmitry
and ask for input on what you need.
I'm happy to work as closely as I can.

Christian

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: glibc at the Toolchains microconference at LPC 2019
  2019-06-26 20:39         ` Christian Brauner
@ 2019-06-26 20:58           ` Carlos O'Donell
  0 siblings, 0 replies; 38+ messages in thread
From: Carlos O'Donell @ 2019-06-26 20:58 UTC (permalink / raw)
  To: Christian Brauner, libc-alpha, Florian Weimer, Zack Weinberg
  Cc: Dmitry V. Levin

On 6/26/19 4:39 PM, Christian Brauner wrote:
> Difficult to get reviewers in the sense of kernel people who wrote
> the syscalls? I'm trying as hard as I can to bridge the libc/kernel
> barrier by always cc'ing e.g. Florian or Dmitry and ask for input on
> what you need. I'm happy to work as closely as I can.

Anyone can review patches on libc-alpha, and provide a "Reviewed-by"
sign-off. With enough of these the patch gains consensus that the right
people have agreed to it. This make subsequent review even easier if
I know the original author of the syscall signs off on the glibc
implementation as meeting their expectations.

For syscall patches we generally want a peer review from another glibc
developer who knows the interfaces and is looking for any mistakes,
or missing information e.g. versions, internal macro usage, docs,
test case usage etc.

So in summary:
- A nod from the original syscall author that their expectations are
  translated into userspace.
- A nod from another glibc developer as a "belt-and-suspenders" that
  the patch doens't have any problems.

That's enough for a commit to master in my book.

-- 
Cheers,
Carlos.

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: glibc at the Toolchains microconference at LPC 2019
  2019-06-26 17:41     ` Christian Brauner
@ 2019-06-26 21:04       ` Carlos O'Donell
  2019-06-27  9:39         ` Dmitry V. Levin
                           ` (2 more replies)
  0 siblings, 3 replies; 38+ messages in thread
From: Carlos O'Donell @ 2019-06-26 21:04 UTC (permalink / raw)
  To: Christian Brauner, libc-alpha, Zack Weinberg, Dmitry V. Levin

On 6/26/19 1:41 PM, Christian Brauner wrote:
> On June 26, 2019 6:50:23 PM GMT+02:00, Zack Weinberg <zackw@panix.com> wrote:
>> On Wed, Jun 26, 2019 at 12:39 PM Dmitry V. Levin <ldv@altlinux.org>
>> wrote:
>>> On Wed, Jun 26, 2019 at 06:01:28PM +0200, Florian Weimer wrote:
>>>> glibc system call wrappers are on the agenda:
>>>>
>>>>
>> <https://www.linuxplumbersconf.org/blog/2019/toolchains-microconference-accepted-into-2019-linux-plumbers-conference/>
>>>>
>>>> Will anyone from the glibc community attend and can set the right
>>>> expectations?
>>>
>>> What are the right expectations?
>>
>> Well, _I_ think glibc should provide wrappers for all Linux system
>> calls, except those that cannot be used without stomping on internal
>> glibc data structures (e.g. set_tid_address, set_robust_list, brk) and
>> those that have been completely superseded by newer syscalls.  Other
>> people have disagreed with me pretty strenuously, but they haven't
>> done any of the work required to make forward progress on their more
>> conservative policies.  I am tempted to post a patch early in the 2.31
>> cycle that adds wrappers for everything, and then threaten to apply it
>> unilaterally unless I hear concrete objections within a week or so.
>>
>> zw
> 
> Strongly agree.
> 

Zach,

Could you please review the language here:
https://sourceware.org/glibc/wiki/Consensus#WIP:_Kernel_syscalls_wrappers

I drafted it in 2014-10-23 based on comments from Joseph Myers and the
community (almost 5 years ago!)

If you agree to that langauge I'll propose this again as being accepted
consensus.

-- 
Cheers,
Carlos.

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: glibc at the Toolchains microconference at LPC 2019
  2019-06-26 21:04       ` Carlos O'Donell
@ 2019-06-27  9:39         ` Dmitry V. Levin
  2019-06-27 10:05           ` Szabolcs Nagy
  2019-06-27 15:57           ` Zack Weinberg
  2019-06-27 13:22         ` glibc at the Toolchains microconference at LPC 2019 Zack Weinberg
  2019-06-27 14:10         ` syscall wrappers policy (was re: glibc at the Toolchains microconference) Zack Weinberg
  2 siblings, 2 replies; 38+ messages in thread
From: Dmitry V. Levin @ 2019-06-27  9:39 UTC (permalink / raw)
  To: Carlos O'Donell; +Cc: Christian Brauner, libc-alpha, Zack Weinberg

[-- Attachment #1: Type: text/plain, Size: 713 bytes --]

On Wed, Jun 26, 2019 at 05:04:52PM -0400, Carlos O'Donell wrote:
[...]
> Could you please review the language here:
> https://sourceware.org/glibc/wiki/Consensus#WIP:_Kernel_syscalls_wrappers

I suggest adding that there is no need to add wrappers for those syscalls
that already have dedicated libraries.

For example, such multiplexers as bpf(2) and keyctl(2) already have
dedicated libraries (libbpf and libkeyutils, respectively) that provide
APIs on top of these raw syscalls.

keyctl(2) manual page explicitly says that "rather than using this system
call directly, you probably want to use the various library functions
mentioned in the descriptions of individual operations".


-- 
ldv

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: glibc at the Toolchains microconference at LPC 2019
  2019-06-27  9:39         ` Dmitry V. Levin
@ 2019-06-27 10:05           ` Szabolcs Nagy
  2019-06-27 10:18             ` Christian Brauner
  2019-06-27 13:05             ` Dmitry V. Levin
  2019-06-27 15:57           ` Zack Weinberg
  1 sibling, 2 replies; 38+ messages in thread
From: Szabolcs Nagy @ 2019-06-27 10:05 UTC (permalink / raw)
  To: Dmitry V. Levin, Carlos O'Donell
  Cc: nd, Christian Brauner, libc-alpha@sourceware.org, Zack Weinberg

On 27/06/2019 10:39, Dmitry V. Levin wrote:
> On Wed, Jun 26, 2019 at 05:04:52PM -0400, Carlos O'Donell wrote:
> [...]
>> Could you please review the language here:
>> https://sourceware.org/glibc/wiki/Consensus#WIP:_Kernel_syscalls_wrappers
> 
> I suggest adding that there is no need to add wrappers for those syscalls
> that already have dedicated libraries.
> 
> For example, such multiplexers as bpf(2) and keyctl(2) already have
> dedicated libraries (libbpf and libkeyutils, respectively) that provide
> APIs on top of these raw syscalls.

there are many issues doing raw syscalls e.g.
the x32 type mess or cancellation support.

external library projects can have different level
of quality, supported abis, header conformance,
security process etc. and they almost always mix
libc and linux uapi headers and types.

so i'm against relying on external libraries
doing raw syscalls (they may provide additional
functionality but the syscall itself should
be in libc)

> 
> keyctl(2) manual page explicitly says that "rather than using this system
> call directly, you probably want to use the various library functions
> mentioned in the descriptions of individual operations".
> 
> 


^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: glibc at the Toolchains microconference at LPC 2019
  2019-06-27 10:05           ` Szabolcs Nagy
@ 2019-06-27 10:18             ` Christian Brauner
  2019-06-27 13:05             ` Dmitry V. Levin
  1 sibling, 0 replies; 38+ messages in thread
From: Christian Brauner @ 2019-06-27 10:18 UTC (permalink / raw)
  To: Szabolcs Nagy
  Cc: Dmitry V. Levin, Carlos O'Donell, nd,
	libc-alpha@sourceware.org, Zack Weinberg

On Thu, Jun 27, 2019 at 10:05:24AM +0000, Szabolcs Nagy wrote:
> On 27/06/2019 10:39, Dmitry V. Levin wrote:
> > On Wed, Jun 26, 2019 at 05:04:52PM -0400, Carlos O'Donell wrote:
> > [...]
> >> Could you please review the language here:
> >> https://sourceware.org/glibc/wiki/Consensus#WIP:_Kernel_syscalls_wrappers
> > 
> > I suggest adding that there is no need to add wrappers for those syscalls
> > that already have dedicated libraries.
> > 
> > For example, such multiplexers as bpf(2) and keyctl(2) already have
> > dedicated libraries (libbpf and libkeyutils, respectively) that provide
> > APIs on top of these raw syscalls.
> 
> there are many issues doing raw syscalls e.g.
> the x32 type mess or cancellation support.
> 
> external library projects can have different level
> of quality, supported abis, header conformance,
> security process etc. and they almost always mix
> libc and linux uapi headers and types.
> 
> so i'm against relying on external libraries
> doing raw syscalls (they may provide additional
> functionality but the syscall itself should
> be in libc)

I agree.
Not just does the quality of those libraries often vary wildly.
Sometimes there are competing libraries and it is unclear which one is
properly maintained and which one is not. (A good example are the
capability libraries libcap2 and libcap-ng (or whatever it's called).)

I think that (g)libc should not make a judgement on whether or not a
syscall should be exposed apart from the caveats Carlos and Zack already
pointed out. It's a massive paintpoint for userspace if the only options
are: link against a (possibly unmaintained) library or do the raw
syscall yourself.

Christian

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: glibc at the Toolchains microconference at LPC 2019
  2019-06-27 10:05           ` Szabolcs Nagy
  2019-06-27 10:18             ` Christian Brauner
@ 2019-06-27 13:05             ` Dmitry V. Levin
  1 sibling, 0 replies; 38+ messages in thread
From: Dmitry V. Levin @ 2019-06-27 13:05 UTC (permalink / raw)
  To: Szabolcs Nagy
  Cc: Carlos O'Donell, nd, Christian Brauner, Zack Weinberg,
	libc-alpha

[-- Attachment #1: Type: text/plain, Size: 1652 bytes --]

On Thu, Jun 27, 2019 at 10:05:24AM +0000, Szabolcs Nagy wrote:
> On 27/06/2019 10:39, Dmitry V. Levin wrote:
> > On Wed, Jun 26, 2019 at 05:04:52PM -0400, Carlos O'Donell wrote:
> > [...]
> >> Could you please review the language here:
> >> https://sourceware.org/glibc/wiki/Consensus#WIP:_Kernel_syscalls_wrappers
> > 
> > I suggest adding that there is no need to add wrappers for those syscalls
> > that already have dedicated libraries.
> > 
> > For example, such multiplexers as bpf(2) and keyctl(2) already have
> > dedicated libraries (libbpf and libkeyutils, respectively) that provide
> > APIs on top of these raw syscalls.
> 
> there are many issues doing raw syscalls e.g.
> the x32 type mess or cancellation support.

Yet raw syscalls have always been in use and this is not going to change.
If there are issues, we should consider providing appropriate interfaces
for invoking raw syscalls without these issues.

> external library projects can have different level
> of quality, supported abis, header conformance,
> security process etc. and they almost always mix
> libc and linux uapi headers and types.

This is all true but ...

> so i'm against relying on external libraries
> doing raw syscalls (they may provide additional
> functionality but the syscall itself should
> be in libc)

... some syscalls seem to have interfaces explicitly designed to scare
regular users off and encourage them to use library functions instead.
bpf(2) and keyctl(2) are examples of such interfaces, providing glibc
wrappers for them would mislead people into invoking these system calls
directly.


-- 
ldv

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: glibc at the Toolchains microconference at LPC 2019
  2019-06-26 21:04       ` Carlos O'Donell
  2019-06-27  9:39         ` Dmitry V. Levin
@ 2019-06-27 13:22         ` Zack Weinberg
  2019-06-27 14:10         ` syscall wrappers policy (was re: glibc at the Toolchains microconference) Zack Weinberg
  2 siblings, 0 replies; 38+ messages in thread
From: Zack Weinberg @ 2019-06-27 13:22 UTC (permalink / raw)
  To: Carlos O'Donell; +Cc: Christian Brauner, GNU C Library, Dmitry V. Levin

On Wed, Jun 26, 2019 at 5:04 PM Carlos O'Donell <carlos@redhat.com> wrote:
>
> On 6/26/19 1:41 PM, Christian Brauner wrote:
> > On June 26, 2019 6:50:23 PM GMT+02:00, Zack Weinberg <zackw@panix.com> wrote:
> >> On Wed, Jun 26, 2019 at 12:39 PM Dmitry V. Levin <ldv@altlinux.org>
> >> wrote:
> >>> On Wed, Jun 26, 2019 at 06:01:28PM +0200, Florian Weimer wrote:
> >>>> glibc system call wrappers are on the agenda:
> >>>>
> >>>>
> >> <https://www.linuxplumbersconf.org/blog/2019/toolchains-microconference-accepted-into-2019-linux-plumbers-conference/>
> >>>>
> >>>> Will anyone from the glibc community attend and can set the right
> >>>> expectations?
> >>>
> >>> What are the right expectations?
> >>
> >> Well, _I_ think glibc should provide wrappers for all Linux system
> >> calls, except those that cannot be used without stomping on internal
> >> glibc data structures (e.g. set_tid_address, set_robust_list, brk) and
> >> those that have been completely superseded by newer syscalls.  Other
> >> people have disagreed with me pretty strenuously, but they haven't
> >> done any of the work required to make forward progress on their more
> >> conservative policies.  I am tempted to post a patch early in the 2.31
> >> cycle that adds wrappers for everything, and then threaten to apply it
> >> unilaterally unless I hear concrete objections within a week or so.
> >>
> >> zw
> >
> > Strongly agree.
> >
>
> Zach,
>
> Could you please review the language here:
> https://sourceware.org/glibc/wiki/Consensus#WIP:_Kernel_syscalls_wrappers
>
> I drafted it in 2014-10-23 based on comments from Joseph Myers and the
> community (almost 5 years ago!)
>
> If you agree to that langauge I'll propose this again as being accepted
> consensus.
>
> --
> Cheers,
> Carlos.

^ permalink raw reply	[flat|nested] 38+ messages in thread

* syscall wrappers policy (was re: glibc at the Toolchains microconference)
  2019-06-26 21:04       ` Carlos O'Donell
  2019-06-27  9:39         ` Dmitry V. Levin
  2019-06-27 13:22         ` glibc at the Toolchains microconference at LPC 2019 Zack Weinberg
@ 2019-06-27 14:10         ` Zack Weinberg
  2019-06-27 15:48           ` DJ Delorie
  2019-06-27 16:50           ` Carlos O'Donell
  2 siblings, 2 replies; 38+ messages in thread
From: Zack Weinberg @ 2019-06-27 14:10 UTC (permalink / raw)
  To: Carlos O'Donell; +Cc: GNU C Library

[sorry about the blank reply earlier, I pushed the wrong button]

On Wed, Jun 26, 2019 at 5:04 PM Carlos O'Donell <carlos@redhat.com> wrote:
>
> Could you please review the language here:
> https://sourceware.org/glibc/wiki/Consensus#WIP:_Kernel_syscalls_wrappers
>
> I drafted it in 2014-10-23 based on comments from Joseph Myers and the
> community (almost 5 years ago!)

This seems mostly right to me, but I have some concerns.

First, I think we need a definition of “syscall wrapper.”  Proposed: A
syscall wrapper is a function that, on a system with a traditional
Unix-style kernel (i.e. we don’t look at Hurd when assessing this)
does substantially all of its work by calling into the kernel.  It may
shuffle its arguments around, it may translate between the glibc ABI
and some lower-level ABI, and it may not actually perform a
system-call trap (e.g. `clock_gettime`), but it couldn’t do what it
does without invoking kernel code, and it doesn’t do any nontrivial
work itself.

The “nontrivial work itself” part is meant to draw a line with `open`
and `signal` on the “is a syscall wrapper” side (even though, on
modern systems, the “true” system calls they use are named `openat`
and `rt_sigaction` respectively), and `pthread_mutex_lock` on the
“isn’t a syscall wrapper” side (even though it does make system calls
under some circumstances, it does a lot of work itself).  I would like
to tighten this up but I’m not having any luck thinking of a better way
to phrase it.

Second, I think we need to talk a bit about the rationale for the
policy.  Something about how it has traditionally been the C library’s
responsibility to provide minimally mediated access to all of the
functionality of the kernel, and how in the past GNU libc has
hesitated to add Linux-specific syscall wrappers on portability and/or
backward compatibility grounds, but we now think that was a mistake.

> * If a syscall is obsoleted by another syscall (or otherwise
>   considered obsolete), there is no need to add a wrapper to glibc.

In the user-facing documentation (not necessarily the consensus rules)
it might be worth adding some kind of reassurance that we’re not going
to get rid of `open` even though it _is_ traditionally a “syscall
wrapper” and `openat` supersedes it.  “Existing functions that
traditionally were syscall wrappers will be preserved, even if the
syscall that does exactly what they do is obsoleted by a newer one
with broader capabilities, if they are specified by ISO C and/or
POSIX, or if they are widely used” or something like that.

But, at the same time, we _do_ need to say that syscall wrappers that
are OS-specific are subject to weaker compatibility guarantees.  We
have, after all, been removing things like nfsservctl and ustat.

> * If a syscall cannot meaningfully be used behind glibc's back, or
>   is not useful in the glibc context except for in the ways in which
>   it is used by glibc

I would prefer to narrow this to something like what I said in my
previous message: “glibc will not provide wrappers for syscalls that
are _impossible_ to use from a program linked against glibc without
corrupting glibc’s internal data structures.  For instance,
`set_thread_area`, `set_tid_address`, `set_robust_list`.”
“Meaningfully used” and “not useful in the glibc context” are too
fuzzy and I’m worried we will be setting ourselves up for arguments.

> * If there's a glibc function that's not quite a direct wrapper of
>   the syscall but provides all the functionality of it that can
>   usefully be used in a program using glibc, there is no need
> * Wrappers should be added … unless there is a clear reason not to

I do not understand the rationale for these exceptions.  Did you have
specific cases in mind when you wrote these?

(I’m particularly concerned that the “not quite a direct wrapper” rule
would be used to argue against exposing a variant of `clone` that
returns twice like `fork` does, which is a thing I think we should
have.  You probably _can_ do any fork-with-options operation with the
`clone` wrapper we have, but having to separate the child-side code to
its own function and allocate stack space for it can be a serious obstacle.)

zw

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: syscall wrappers policy (was re: glibc at the Toolchains microconference)
  2019-06-27 14:10         ` syscall wrappers policy (was re: glibc at the Toolchains microconference) Zack Weinberg
@ 2019-06-27 15:48           ` DJ Delorie
  2019-06-27 16:02             ` Zack Weinberg
  2019-06-27 16:50           ` Carlos O'Donell
  1 sibling, 1 reply; 38+ messages in thread
From: DJ Delorie @ 2019-06-27 15:48 UTC (permalink / raw)
  To: Zack Weinberg; +Cc: carlos, libc-alpha

Zack Weinberg <zackw@panix.com> writes:
> First, I think we need a definition of “syscall wrapper.”  Proposed:

I offer alternate wording, not because I think mine is better, but to
stir the pot a bit and get people thinking :-)

"A syscall wrapper is a function whose primary (preferably sole) purpose
is to provide a minimal C-callable interface to a kernel function."

I think that explains why glibc has them (C-callable), limits its scope
(primary/sole purpose) and defines its purpose (kernel function).

How glibc implements them, and how much it needs to do to retain
compatiblity with other C APIs and standards, can be documented
separately.  Or we could add ", consistent with other glibc functions"
to the end of the above.

> Second, I think we need to talk a bit about the rationale for the
> policy.

I think definining it as a "C API" makes it clear that the C library is
the right place for C functions.  That also keeps us from trying to be
the ONLY kernel API library, when other languages need non-C wrappers.

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: glibc at the Toolchains microconference at LPC 2019
  2019-06-27  9:39         ` Dmitry V. Levin
  2019-06-27 10:05           ` Szabolcs Nagy
@ 2019-06-27 15:57           ` Zack Weinberg
  2019-06-27 17:21             ` Florian Weimer
  2019-06-27 20:53             ` wrappers for multiplexed syscalls (was Re: glibc at the Toolchains microconference at LPC 2019) Dmitry V. Levin
  1 sibling, 2 replies; 38+ messages in thread
From: Zack Weinberg @ 2019-06-27 15:57 UTC (permalink / raw)
  To: Dmitry V. Levin; +Cc: Carlos O'Donell, GNU C Library

On Thu, Jun 27, 2019 at 5:39 AM Dmitry V. Levin <ldv@altlinux.org> wrote:
>
> On Wed, Jun 26, 2019 at 05:04:52PM -0400, Carlos O'Donell wrote:
> [...]
> > Could you please review the language here:
> > https://sourceware.org/glibc/wiki/Consensus#WIP:_Kernel_syscalls_wrappers
>
> I suggest adding that there is no need to add wrappers for those syscalls
> that already have dedicated libraries.
>
> For example, such multiplexers as bpf(2) and keyctl(2) already have
> dedicated libraries (libbpf and libkeyutils, respectively) that provide
> APIs on top of these raw syscalls.

I specifically disagree with this.  The existence of these dedicated
libraries does not mean that there is no need for a minimal wrapper in
the C library.  In fact, providing a minimal wrapper in the C library
would make the implementation of dedicated libraries easier, since
they can concentrate on designing their higher-level API rather than
wasting engineering effort on system call wrappers.  glibc has already
done all of the low-level work necessary.

I am a little disappointed to see that Linux is still inventing new
multiplexed system calls, though.  I thought that was demonstrated to
be a bad idea back in the days of __NR_ipc.

zw

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: syscall wrappers policy (was re: glibc at the Toolchains microconference)
  2019-06-27 15:48           ` DJ Delorie
@ 2019-06-27 16:02             ` Zack Weinberg
  0 siblings, 0 replies; 38+ messages in thread
From: Zack Weinberg @ 2019-06-27 16:02 UTC (permalink / raw)
  To: DJ Delorie; +Cc: Carlos O'Donell, GNU C Library

On Thu, Jun 27, 2019 at 11:49 AM DJ Delorie <dj@redhat.com> wrote:
>
> Zack Weinberg <zackw@panix.com> writes:
> > First, I think we need a definition of “syscall wrapper.”  Proposed:
>
> I offer alternate wording, not because I think mine is better, but to
> stir the pot a bit and get people thinking :-)
>
> "A syscall wrapper is a function whose primary (preferably sole) purpose
> is to provide a minimal C-callable interface to a kernel function."
>
> I think that explains why glibc has them (C-callable), limits its scope
> (primary/sole purpose) and defines its purpose (kernel function).

I like this.  "Minimal interface" is better than "doesn't do any
nontrivial work itself."

> > Second, I think we need to talk a bit about the rationale for the
> > policy.
>
> I think definining it as a "C API" makes it clear that the C library is
> the right place for C functions.  That also keeps us from trying to be
> the ONLY kernel API library, when other languages need non-C wrappers.

Yeah.  I feel like there's a need, nowadays, for a language-agnostic
kernel API library and a dynamic loader completely decoupled from the
C library, but that's a big can of worms and we shouldn't let it get
in the way of anything else.

zw

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: syscall wrappers policy (was re: glibc at the Toolchains microconference)
  2019-06-27 14:10         ` syscall wrappers policy (was re: glibc at the Toolchains microconference) Zack Weinberg
  2019-06-27 15:48           ` DJ Delorie
@ 2019-06-27 16:50           ` Carlos O'Donell
  1 sibling, 0 replies; 38+ messages in thread
From: Carlos O'Donell @ 2019-06-27 16:50 UTC (permalink / raw)
  To: Zack Weinberg; +Cc: GNU C Library

On 6/27/19 10:10 AM, Zack Weinberg wrote:
>> * Wrappers should be added … unless there is a clear reason not to
> 
> I do not understand the rationale for these exceptions.  Did you have
> specific cases in mind when you wrote these?

I think it was just a general escape hatch which is not needed.

If we find a really bad example then we'll have a public discussion about
it and can use consensus to change the rules.

Therefore the rules as written can be as strict as we agree to.

> (I’m particularly concerned that the “not quite a direct wrapper” rule
> would be used to argue against exposing a variant of `clone` that
> returns twice like `fork` does, which is a thing I think we should
> have.  You probably _can_ do any fork-with-options operation with the
> `clone` wrapper we have, but having to separate the child-side code to
> its own function and allocate stack space for it can be a serious obstacle.)

Updated. Please review.

https://sourceware.org/glibc/wiki/Consensus#WIP:_Kernel_syscalls_wrappers

-- 
Cheers,
Carlos.

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: glibc at the Toolchains microconference at LPC 2019
  2019-06-27 15:57           ` Zack Weinberg
@ 2019-06-27 17:21             ` Florian Weimer
  2019-06-27 20:07               ` Carlos O'Donell
  2019-06-27 20:53             ` wrappers for multiplexed syscalls (was Re: glibc at the Toolchains microconference at LPC 2019) Dmitry V. Levin
  1 sibling, 1 reply; 38+ messages in thread
From: Florian Weimer @ 2019-06-27 17:21 UTC (permalink / raw)
  To: Zack Weinberg; +Cc: Dmitry V. Levin, Carlos O'Donell, GNU C Library

* Zack Weinberg:

> I specifically disagree with this.  The existence of these dedicated
> libraries does not mean that there is no need for a minimal wrapper in
> the C library.  In fact, providing a minimal wrapper in the C library
> would make the implementation of dedicated libraries easier, since
> they can concentrate on designing their higher-level API rather than
> wasting engineering effort on system call wrappers.  glibc has already
> done all of the low-level work necessary.

We would have to begin backporting syscall wrappers, though.  Otherwise
these libraries are blocked until a glibc upgrade, which may not happen
any time soon.

Maybe we can move well-established libraries into glibc eventually, but
that can have unpredictable results if those libraries did not use
symbol versioning from the start (so that their implementation
interposes a newer glibc implementation for the entire process).

But I don't know if this (no syscall wrappers except in glibc) works as
a default policy.

Thanks,
Florian

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: glibc at the Toolchains microconference at LPC 2019
  2019-06-26 16:01 glibc at the Toolchains microconference at LPC 2019 Florian Weimer
  2019-06-26 16:39 ` Dmitry V. Levin
@ 2019-06-27 17:45 ` Maciej Rozycki
  2019-07-08 12:57   ` Florian Weimer
  1 sibling, 1 reply; 38+ messages in thread
From: Maciej Rozycki @ 2019-06-27 17:45 UTC (permalink / raw)
  To: Florian Weimer; +Cc: libc-alpha@sourceware.org

Hi Florian,

> glibc system call wrappers are on the agenda:
> 
> <https://www.linuxplumbersconf.org/blog/2019/toolchains-microconference-accepted-into-2019-linux-plumbers-conference/>
> 
> Will anyone from the glibc community attend and can set the right
> expectations?

 I plan to be there and I can carry the message.

> I won't be there because I'm traveling to Cauldron instead, which is
> immediately after LPC, but in a different corner of the planet.

 I actually plan to be there at both events, so I can bring back to the 
Cauldron whatever we'll have come up with at LPC too.

  Maciej

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: glibc at the Toolchains microconference at LPC 2019
  2019-06-27 17:21             ` Florian Weimer
@ 2019-06-27 20:07               ` Carlos O'Donell
  0 siblings, 0 replies; 38+ messages in thread
From: Carlos O'Donell @ 2019-06-27 20:07 UTC (permalink / raw)
  To: Florian Weimer, Zack Weinberg; +Cc: Dmitry V. Levin, GNU C Library

On 6/27/19 1:21 PM, Florian Weimer wrote:
> * Zack Weinberg:
> 
>> I specifically disagree with this.  The existence of these dedicated
>> libraries does not mean that there is no need for a minimal wrapper in
>> the C library.  In fact, providing a minimal wrapper in the C library
>> would make the implementation of dedicated libraries easier, since
>> they can concentrate on designing their higher-level API rather than
>> wasting engineering effort on system call wrappers.  glibc has already
>> done all of the low-level work necessary.
> 
> We would have to begin backporting syscall wrappers, though.  Otherwise
> these libraries are blocked until a glibc upgrade, which may not happen
> any time soon.

Correct.

> Maybe we can move well-established libraries into glibc eventually, but
> that can have unpredictable results if those libraries did not use
> symbol versioning from the start (so that their implementation
> interposes a newer glibc implementation for the entire process).

I don't think anyone is asking for this, but it will be a question that
comes up when established practice is migrated to glibc.

How did libstdc++'s adoption of boost APIs handle this? I think they
just renamed things?

> But I don't know if this (no syscall wrappers except in glibc) works as
> a default policy.

I didn't read Zack's comment as meaning "no syscall wrappers *except* in glibc,"
but "users can trust glibc to *always* provide syscall wrappers" in that we
as a community should not shirk our duty to provide syscall wrappers because
project X, Y, or Z says they will do it for us or are already doing it as
part of some userspace interface. There will always be users that want that
minimal C-callable interface.

-- 
Cheers,
Carlos.

^ permalink raw reply	[flat|nested] 38+ messages in thread

* wrappers for multiplexed syscalls (was Re: glibc at the Toolchains microconference at LPC 2019)
  2019-06-27 15:57           ` Zack Weinberg
  2019-06-27 17:21             ` Florian Weimer
@ 2019-06-27 20:53             ` Dmitry V. Levin
  2019-06-27 21:22               ` Florian Weimer
  1 sibling, 1 reply; 38+ messages in thread
From: Dmitry V. Levin @ 2019-06-27 20:53 UTC (permalink / raw)
  To: Zack Weinberg; +Cc: Carlos O'Donell, GNU C Library

[-- Attachment #1: Type: text/plain, Size: 2086 bytes --]

On Thu, Jun 27, 2019 at 11:57:46AM -0400, Zack Weinberg wrote:
> On Thu, Jun 27, 2019 at 5:39 AM Dmitry V. Levin wrote:
> >
> > On Wed, Jun 26, 2019 at 05:04:52PM -0400, Carlos O'Donell wrote:
> > [...]
> > > Could you please review the language here:
> > > https://sourceware.org/glibc/wiki/Consensus#WIP:_Kernel_syscalls_wrappers
> >
> > I suggest adding that there is no need to add wrappers for those syscalls
> > that already have dedicated libraries.
> >
> > For example, such multiplexers as bpf(2) and keyctl(2) already have
> > dedicated libraries (libbpf and libkeyutils, respectively) that provide
> > APIs on top of these raw syscalls.
> 
> I specifically disagree with this.  The existence of these dedicated
> libraries does not mean that there is no need for a minimal wrapper in
> the C library.  In fact, providing a minimal wrapper in the C library
> would make the implementation of dedicated libraries easier, since
> they can concentrate on designing their higher-level API rather than
> wasting engineering effort on system call wrappers.  glibc has already
> done all of the low-level work necessary.
> 
> I am a little disappointed to see that Linux is still inventing new
> multiplexed system calls, though.  I thought that was demonstrated to
> be a bad idea back in the days of __NR_ipc.

Speaking of multiplexed syscalls, if we don't event contemplate an idea
of adding a glibc wrapper for __NR_ipc, why would we want to add
a wrapper for __NR_bpf or __NR_keyctl?

Since these syscalls have interfaces explicitly designed to scare regular
users off and encourage them to use library functions instead, providing
glibc wrappers for them would mislead people into invoking these system
calls directly.  These wrappers would lack type checking available in
higher-level library functions, causing unnecessary runtime errors that
are easily avoidable at compile time when higher-level library functions
are used.

Can you demonstrate any potential usefulness of glibc wrappers for
__NR_bpf and __NR_keyctl?


-- 
ldv

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: wrappers for multiplexed syscalls (was Re: glibc at the Toolchains microconference at LPC 2019)
  2019-06-27 20:53             ` wrappers for multiplexed syscalls (was Re: glibc at the Toolchains microconference at LPC 2019) Dmitry V. Levin
@ 2019-06-27 21:22               ` Florian Weimer
  2019-06-28 21:20                 ` Dmitry V. Levin
  0 siblings, 1 reply; 38+ messages in thread
From: Florian Weimer @ 2019-06-27 21:22 UTC (permalink / raw)
  To: Dmitry V. Levin; +Cc: Zack Weinberg, Carlos O'Donell, GNU C Library

* Dmitry V. Levin:

> Speaking of multiplexed syscalls, if we don't event contemplate an idea
> of adding a glibc wrapper for __NR_ipc, 

I think someonme did argue for socketcall when the topic came up a
while back.

> why would we want to add a wrapper for __NR_bpf or __NR_keyctl?

We have wrappers for ioctl, fcntl, ptrace, prctl.  I suppose they
could serve as models.

We'll see what the future will bring on the Linux.  I expect that
system calls are now easier to wire up consistently across
architectures, so perhaps we will see fewer multiplexer system calls
in the future.  I suspect that, for architecture-independent features,
these multiplexers had the important property that a kernel
contributor wouldn't need to touch arch-specific code to add further
sub-calls, which made them very attractive if you want to roll out new
features relatively quickly.

Regarding fcntl, I did look into providing a type-safe version of it
using _Generic/__builtin_types_compatible_p/__builtin_choose_expr and
so on.  But then I realized that instead, we probably should have
separate fcntl_* functions for each sub-call, similar to what we did
for socketcall.  gnulib could easily provide high-quality
implementations for libcs which do not pick up on this idea.

So yes, I tend to agree that exposing wrappers is problematic.

> Since these syscalls have interfaces explicitly designed to scare regular
> users off and encourage them to use library functions instead, providing
> glibc wrappers for them would mislead people into invoking these system
> calls directly.  These wrappers would lack type checking available in
> higher-level library functions, causing unnecessary runtime errors that
> are easily avoidable at compile time when higher-level library functions
> are used.

Right.  And fcntl isn't so different, actually.

Thanks,
Florian

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: wrappers for multiplexed syscalls (was Re: glibc at the Toolchains microconference at LPC 2019)
  2019-06-27 21:22               ` Florian Weimer
@ 2019-06-28 21:20                 ` Dmitry V. Levin
  2019-06-28 21:35                   ` Florian Weimer
  0 siblings, 1 reply; 38+ messages in thread
From: Dmitry V. Levin @ 2019-06-28 21:20 UTC (permalink / raw)
  To: Florian Weimer; +Cc: Zack Weinberg, Carlos O'Donell, GNU C Library

[-- Attachment #1: Type: text/plain, Size: 2300 bytes --]

On Thu, Jun 27, 2019 at 11:22:45PM +0200, Florian Weimer wrote:
> * Dmitry V. Levin:
> > why would we want to add a wrapper for __NR_bpf or __NR_keyctl?
> 
> We have wrappers for ioctl, fcntl, ptrace, prctl.  I suppose they
> could serve as models.
> 
> We'll see what the future will bring on the Linux.  I expect that
> system calls are now easier to wire up consistently across
> architectures, so perhaps we will see fewer multiplexer system calls
> in the future.  I suspect that, for architecture-independent features,
> these multiplexers had the important property that a kernel
> contributor wouldn't need to touch arch-specific code to add further
> sub-calls, which made them very attractive if you want to roll out new
> features relatively quickly.
> 
> Regarding fcntl, I did look into providing a type-safe version of it
> using _Generic/__builtin_types_compatible_p/__builtin_choose_expr and
> so on.  But then I realized that instead, we probably should have
> separate fcntl_* functions for each sub-call, similar to what we did
> for socketcall.  gnulib could easily provide high-quality
> implementations for libcs which do not pick up on this idea.
> 
> So yes, I tend to agree that exposing wrappers is problematic.
> 
> > Since these syscalls have interfaces explicitly designed to scare regular
> > users off and encourage them to use library functions instead, providing
> > glibc wrappers for them would mislead people into invoking these system
> > calls directly.  These wrappers would lack type checking available in
> > higher-level library functions, causing unnecessary runtime errors that
> > are easily avoidable at compile time when higher-level library functions
> > are used.
> 
> Right.  And fcntl isn't so different, actually.

What if we start adding separate functions for new interfaces of already
existing multiplexed system call wrappers?

For example, ptrace is going to gain a new command (PTRACE_GET_SYSCALL_INFO)
in Linux 5.3.  My initial plan was just to update sys/ptrace.h with a new
constant and bits/ptrace-shared.h with a new structure, but I could add
a new function as well:

extern int ptrace_get_syscall_info (__pid_t __pid,
				    struct __ptrace_syscall_info *__infop)
	__THROW;


-- 
ldv

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: wrappers for multiplexed syscalls (was Re: glibc at the Toolchains microconference at LPC 2019)
  2019-06-28 21:20                 ` Dmitry V. Levin
@ 2019-06-28 21:35                   ` Florian Weimer
  2019-06-29  1:52                     ` Zack Weinberg
  0 siblings, 1 reply; 38+ messages in thread
From: Florian Weimer @ 2019-06-28 21:35 UTC (permalink / raw)
  To: Dmitry V. Levin; +Cc: Zack Weinberg, Carlos O'Donell, GNU C Library

* Dmitry V. Levin:

> What if we start adding separate functions for new interfaces of already
> existing multiplexed system call wrappers?
>
> For example, ptrace is going to gain a new command (PTRACE_GET_SYSCALL_INFO)
> in Linux 5.3.  My initial plan was just to update sys/ptrace.h with a new
> constant and bits/ptrace-shared.h with a new structure, but I could add
> a new function as well:
>
> extern int ptrace_get_syscall_info (__pid_t __pid,
> 				    struct __ptrace_syscall_info *__infop)
> 	__THROW;

I think this makes a lot of sense.  We should also do this for existing
constants (and for fcntl, too).  Perhaps after we have added more of the
wrappers that are *completely* missing, though.

For futex, it will also help with the time_t transition.

Thanks,
Florian

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: wrappers for multiplexed syscalls (was Re: glibc at the Toolchains microconference at LPC 2019)
  2019-06-28 21:35                   ` Florian Weimer
@ 2019-06-29  1:52                     ` Zack Weinberg
  2019-06-29  6:27                       ` Florian Weimer
  0 siblings, 1 reply; 38+ messages in thread
From: Zack Weinberg @ 2019-06-29  1:52 UTC (permalink / raw)
  To: Florian Weimer; +Cc: Dmitry V. Levin, Carlos O'Donell, GNU C Library

On Fri, Jun 28, 2019 at 5:35 PM Florian Weimer <fweimer@redhat.com> wrote:
>
> * Dmitry V. Levin:
>
> > What if we start adding separate functions for new interfaces of already
> > existing multiplexed system call wrappers?
> >
> > For example, ptrace is going to gain a new command (PTRACE_GET_SYSCALL_INFO)
> > in Linux 5.3.  My initial plan was just to update sys/ptrace.h with a new
> > constant and bits/ptrace-shared.h with a new structure, but I could add
> > a new function as well:
> >
> > extern int ptrace_get_syscall_info (__pid_t __pid,
> >                                   struct __ptrace_syscall_info *__infop)
> >       __THROW;
>
> I think this makes a lot of sense.  We should also do this for existing
> constants (and for fcntl, too).  Perhaps after we have added more of the
> wrappers that are *completely* missing, though.

Yeah, I like this idea as well.  I think we should keep exposing the
multiplexed interfaces, though, just because sometimes they do add new
opcodes (e.g. the OFD locks).

zw

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: wrappers for multiplexed syscalls (was Re: glibc at the Toolchains microconference at LPC 2019)
  2019-06-29  1:52                     ` Zack Weinberg
@ 2019-06-29  6:27                       ` Florian Weimer
  0 siblings, 0 replies; 38+ messages in thread
From: Florian Weimer @ 2019-06-29  6:27 UTC (permalink / raw)
  To: Zack Weinberg; +Cc: Dmitry V. Levin, Carlos O'Donell, GNU C Library

* Zack Weinberg:

> On Fri, Jun 28, 2019 at 5:35 PM Florian Weimer <fweimer@redhat.com> wrote:
>>
>> * Dmitry V. Levin:
>>
>> > What if we start adding separate functions for new interfaces of already
>> > existing multiplexed system call wrappers?
>> >
>> > For example, ptrace is going to gain a new command (PTRACE_GET_SYSCALL_INFO)
>> > in Linux 5.3.  My initial plan was just to update sys/ptrace.h with a new
>> > constant and bits/ptrace-shared.h with a new structure, but I could add
>> > a new function as well:
>> >
>> > extern int ptrace_get_syscall_info (__pid_t __pid,
>> >                                   struct __ptrace_syscall_info *__infop)
>> >       __THROW;
>>
>> I think this makes a lot of sense.  We should also do this for existing
>> constants (and for fcntl, too).  Perhaps after we have added more of the
>> wrappers that are *completely* missing, though.
>
> Yeah, I like this idea as well.  I think we should keep exposing the
> multiplexed interfaces, though, just because sometimes they do add new
> opcodes (e.g. the OFD locks).

Maybe.  But not as varargs functions, so that there's no ambiguity how
many arguments need to be extracted and passed to the system call (which
caused interoperability problems with open/openat).

Thanks,
Florian

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: glibc at the Toolchains microconference at LPC 2019
  2019-06-27 17:45 ` glibc at the Toolchains microconference at LPC 2019 Maciej Rozycki
@ 2019-07-08 12:57   ` Florian Weimer
  2019-07-08 16:06     ` Paul Eggert
                       ` (2 more replies)
  0 siblings, 3 replies; 38+ messages in thread
From: Florian Weimer @ 2019-07-08 12:57 UTC (permalink / raw)
  To: Maciej Rozycki; +Cc: libc-alpha@sourceware.org

* Maciej Rozycki:

>> glibc system call wrappers are on the agenda:
>> 
>> <https://www.linuxplumbersconf.org/blog/2019/toolchains-microconference-accepted-into-2019-linux-plumbers-conference/>
>> 
>> Will anyone from the glibc community attend and can set the right
>> expectations?
>
>  I plan to be there and I can carry the message.

Great, then we need to decide what it would be.

Maybe the following is a reasonable starting point?

The glibc project decided a while back that it wants to add wrappers for
system calls which are useful for general application usage.  However,
that doesn't mean that all those missing system calls are added
immediately.

System call wrappers still need documentation in the manual, which is
can be difficult in areas where there is no consensus how to describe
the desired semantics (e.g., in the area of concurrency).  (Copyright
assignment to the FSF is needed for both the code and the manual update,
but can usually performed electronically these days, and is reasonably
straightforward.)  On top of that, the glibc project is seriously
constrained by available reviewer bandwidth.

Some more specific notes:

Emulation of the system call is not required.  It has been historically
very problematic.  The only thing that has not come back to bite us is
checking if a new flag argument is zero and call the old, equivalent
system call instead in this case.

Wrapper names should be architecture-independent if at all possible.
Sharing system call names as much as possible between architectures in
the UAPI headers helps with that.

Mutiplexing system calls are difficult to wrap, particularly if the
types and number of arguments vary.  Previous attempts to use varargs
for this have lead to bugs.  For example, open/openat would not pass
down the mode flag for O_TMPFILE initially, or cannot be called with a
non-variadic prototype/function pointer on some architectures.  We
wouldn't want to wrap socketcall (even if it had not been superseded),
and may wrap futex as separate functions.

We strongly prefer if a system call that is not inherently
architecture-specific (e.g., some new VFS functionality) is enabled for
all architectures in the same kernel release.

When it comes to exposes the system call, we prefer to use size_t for
buffer sizes (even if the kernel uses int or unsigned int), purely for
documentation purposes.  Flag arguments should not be long int because
it is unclear whether in the future, more than 32 flags will be added on
64-bit architectures.  Except for pthread_* functions, error reporting
is based on errno and special return values.

Passing file offsets through off64_t * arguments is fine with us.
Otherwise, off64_t parameter passing tends to vary too much.

If constants and types related to a particular system call are defined
in a separate header which does not contain much else, we can incldue
that from the glibc headers if available.  As a result, new kernel flags
will become available to application developers immediately once they
install newer kernel headers.  (This may not work for multiplexing
system calls, of course, even if we wrap the multiplexer.)

Any other comments?

Thanks,
Florian

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: glibc at the Toolchains microconference at LPC 2019
  2019-07-08 12:57   ` Florian Weimer
@ 2019-07-08 16:06     ` Paul Eggert
  2019-07-19 11:49       ` Florian Weimer
  2019-07-09  0:14     ` Dmitry V. Levin
  2019-09-16 13:26     ` Christian Brauner
  2 siblings, 1 reply; 38+ messages in thread
From: Paul Eggert @ 2019-07-08 16:06 UTC (permalink / raw)
  To: Florian Weimer, Maciej Rozycki; +Cc: GNU C Library

Florian Weimer wrote:
> When it comes to exposes the system call, we prefer to use size_t for
> buffer sizes (even if the kernel uses int or unsigned int), purely for
> documentation purposes.

I suggest using "ssize_t or size_t" to keep the text parallel with "int or 
unsigned int". We shouldn't insist on size_t for buffer sizes when the natural 
interpretation of the value is signed (as in the return value for 'read' etc.) 
or is too large for malloc etc.

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: glibc at the Toolchains microconference at LPC 2019
  2019-07-08 12:57   ` Florian Weimer
  2019-07-08 16:06     ` Paul Eggert
@ 2019-07-09  0:14     ` Dmitry V. Levin
  2019-09-16 13:26     ` Christian Brauner
  2 siblings, 0 replies; 38+ messages in thread
From: Dmitry V. Levin @ 2019-07-09  0:14 UTC (permalink / raw)
  To: Florian Weimer; +Cc: Maciej Rozycki, libc-alpha

[-- Attachment #1: Type: text/plain, Size: 640 bytes --]

On Mon, Jul 08, 2019 at 02:57:32PM +0200, Florian Weimer wrote:
> * Maciej Rozycki:
> 
> >> glibc system call wrappers are on the agenda:
> >> 
> >> <https://www.linuxplumbersconf.org/blog/2019/toolchains-microconference-accepted-into-2019-linux-plumbers-conference/>
> >> Will anyone from the glibc community attend and can set the right
> >> expectations?
> >
> >  I plan to be there and I can carry the message.
> 
> Great, then we need to decide what it would be.
> 
> Maybe the following is a reasonable starting point?

Looks good to me (modulo typos).
Apparently, I'm going to be at both events, too. 


-- 
ldv

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: glibc at the Toolchains microconference at LPC 2019
  2019-07-08 16:06     ` Paul Eggert
@ 2019-07-19 11:49       ` Florian Weimer
  2019-09-14  4:33         ` Maciej W. Rozycki
  0 siblings, 1 reply; 38+ messages in thread
From: Florian Weimer @ 2019-07-19 11:49 UTC (permalink / raw)
  To: Paul Eggert; +Cc: Maciej Rozycki, GNU C Library

* Paul Eggert:

> Florian Weimer wrote:
>> When it comes to exposes the system call, we prefer to use size_t for
>> buffer sizes (even if the kernel uses int or unsigned int), purely for
>> documentation purposes.
>
> I suggest using "ssize_t or size_t" to keep the text parallel with
> "int or unsigned int". We shouldn't insist on size_t for buffer sizes
> when the natural interpretation of the value is signed (as in the
> return value for 'read' etc.) or is too large for malloc etc.

Right.  Incoming buffer sizes should be size_t (no in-band signaling).
Return values tend to have in-band signaling for errors and thus need to
be ssize_t.

Thanks,
Florian

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: glibc at the Toolchains microconference at LPC 2019
  2019-07-19 11:49       ` Florian Weimer
@ 2019-09-14  4:33         ` Maciej W. Rozycki
  2019-09-14 11:53           ` Dmitry V. Levin
  0 siblings, 1 reply; 38+ messages in thread
From: Maciej W. Rozycki @ 2019-09-14  4:33 UTC (permalink / raw)
  To: Florian Weimer; +Cc: Paul Eggert, GNU C Library

On Fri, 19 Jul 2019, Florian Weimer wrote:

> > I suggest using "ssize_t or size_t" to keep the text parallel with
> > "int or unsigned int". We shouldn't insist on size_t for buffer sizes
> > when the natural interpretation of the value is signed (as in the
> > return value for 'read' etc.) or is too large for malloc etc.
> 
> Right.  Incoming buffer sizes should be size_t (no in-band signaling).
> Return values tend to have in-band signaling for errors and thus need to
> be ssize_t.

 FYI, the response was positive and a rough transcript of the session is 
available from <https://etherpad.net/p/LPC2019_TC>.  I'll be there at the 
glibc BoF available to comment.

  Maciej

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: glibc at the Toolchains microconference at LPC 2019
  2019-09-14  4:33         ` Maciej W. Rozycki
@ 2019-09-14 11:53           ` Dmitry V. Levin
  2019-09-14 14:02             ` Maciej W. Rozycki
  0 siblings, 1 reply; 38+ messages in thread
From: Dmitry V. Levin @ 2019-09-14 11:53 UTC (permalink / raw)
  To: GNU C Library
  Cc: H. Peter Anvin, Maciej W. Rozycki, Florian Weimer, Paul Eggert

[-- Attachment #1: Type: text/plain, Size: 476 bytes --]

On Sat, Sep 14, 2019 at 05:33:03AM +0100, Maciej W. Rozycki wrote:
[...]
>  FYI, the response was positive and a rough transcript of the session is 
> available from <https://etherpad.net/p/LPC2019_TC>.  I'll be there at the 
> glibc BoF available to comment.

I had a short conversation with H. Peter Anvin (Cc'ed) who missed the talk.
Apparently, he wants to proceed with his plan to add a fallback library
with syscall wrappers into the kernel tree.


-- 
ldv

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: glibc at the Toolchains microconference at LPC 2019
  2019-09-14 11:53           ` Dmitry V. Levin
@ 2019-09-14 14:02             ` Maciej W. Rozycki
  0 siblings, 0 replies; 38+ messages in thread
From: Maciej W. Rozycki @ 2019-09-14 14:02 UTC (permalink / raw)
  To: Dmitry V. Levin
  Cc: GNU C Library, H. Peter Anvin, Florian Weimer, Paul Eggert

On Sat, 14 Sep 2019, Dmitry V. Levin wrote:

> I had a short conversation with H. Peter Anvin (Cc'ed) who missed the talk.
> Apparently, he wants to proceed with his plan to add a fallback library
> with syscall wrappers into the kernel tree.

 Oh, great (on both accounts).

  Maciej

^ permalink raw reply	[flat|nested] 38+ messages in thread

* Re: glibc at the Toolchains microconference at LPC 2019
  2019-07-08 12:57   ` Florian Weimer
  2019-07-08 16:06     ` Paul Eggert
  2019-07-09  0:14     ` Dmitry V. Levin
@ 2019-09-16 13:26     ` Christian Brauner
  2 siblings, 0 replies; 38+ messages in thread
From: Christian Brauner @ 2019-09-16 13:26 UTC (permalink / raw)
  To: Florian Weimer, Maciej Rozycki, libc-alpha@sourceware.org

On Mon, Jul 08, 2019 at 02:57:32PM +0200, Florian Weimer wrote:
> * Maciej Rozycki:
> 
> >> glibc system call wrappers are on the agenda:
> >> 
> >> <https://www.linuxplumbersconf.org/blog/2019/toolchains-microconference-accepted-into-2019-linux-plumbers-conference/>

This was an excellent session and I'm quite pleased with the
outcome/agreement.
While we're at it: kcmp() has no glibc syscall wrapper. Would be great
if there would be one. :)

Thanks!
Christian

^ permalink raw reply	[flat|nested] 38+ messages in thread

end of thread, other threads:[~2019-09-16 13:26 UTC | newest]

Thread overview: 38+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-06-26 16:01 glibc at the Toolchains microconference at LPC 2019 Florian Weimer
2019-06-26 16:39 ` Dmitry V. Levin
2019-06-26 16:50   ` Zack Weinberg
2019-06-26 16:56     ` Florian Weimer
2019-06-26 17:40       ` Zack Weinberg
2019-06-26 20:33       ` Carlos O'Donell
2019-06-26 20:39         ` Christian Brauner
2019-06-26 20:58           ` Carlos O'Donell
2019-06-26 17:28     ` Yann Droneaud
2019-06-26 17:41     ` Christian Brauner
2019-06-26 21:04       ` Carlos O'Donell
2019-06-27  9:39         ` Dmitry V. Levin
2019-06-27 10:05           ` Szabolcs Nagy
2019-06-27 10:18             ` Christian Brauner
2019-06-27 13:05             ` Dmitry V. Levin
2019-06-27 15:57           ` Zack Weinberg
2019-06-27 17:21             ` Florian Weimer
2019-06-27 20:07               ` Carlos O'Donell
2019-06-27 20:53             ` wrappers for multiplexed syscalls (was Re: glibc at the Toolchains microconference at LPC 2019) Dmitry V. Levin
2019-06-27 21:22               ` Florian Weimer
2019-06-28 21:20                 ` Dmitry V. Levin
2019-06-28 21:35                   ` Florian Weimer
2019-06-29  1:52                     ` Zack Weinberg
2019-06-29  6:27                       ` Florian Weimer
2019-06-27 13:22         ` glibc at the Toolchains microconference at LPC 2019 Zack Weinberg
2019-06-27 14:10         ` syscall wrappers policy (was re: glibc at the Toolchains microconference) Zack Weinberg
2019-06-27 15:48           ` DJ Delorie
2019-06-27 16:02             ` Zack Weinberg
2019-06-27 16:50           ` Carlos O'Donell
2019-06-27 17:45 ` glibc at the Toolchains microconference at LPC 2019 Maciej Rozycki
2019-07-08 12:57   ` Florian Weimer
2019-07-08 16:06     ` Paul Eggert
2019-07-19 11:49       ` Florian Weimer
2019-09-14  4:33         ` Maciej W. Rozycki
2019-09-14 11:53           ` Dmitry V. Levin
2019-09-14 14:02             ` Maciej W. Rozycki
2019-07-09  0:14     ` Dmitry V. Levin
2019-09-16 13:26     ` Christian Brauner

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).