Re: [libvirt] Fwd: libvirtd failing on MacOS in setgroups

Re: [libvirt] Fwd: libvirtd failing on MacOS in setgroups

[Eric Blake]

[-- Attachment #1.1: Type: text/plain, Size: 1450 bytes --]

[adding gnulib]

On 8/21/19 1:34 PM, Roman Bolshakov wrote:

>>>>> I get the following error when running libvirtd on MacOS as root:
>>>>>
>>>>> 2019-07-11 00:12:33.673+0000: 123145573953536: error :
>>>>> qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU
>>>>> binary /usr/local/bin/qemu-system-x86_64 for probing: libvirt:  error
>>>>> : cannot set supplemental groups: Invalid argument
>>>>

>> +++ b/src/util/virutil.c
>> @@ -1043,6 +1043,7 @@ virSetUIDGID(uid_t uid, gid_t gid, gid_t *groups ATTRIBUTE_UNUSED,
>>      }
>>  
>>  # if HAVE_SETGROUPS
>> +    VIR_DEBUG("setgroups %d max %d", ngroups, NGROUPS_MAX);
>>      if (gid != (gid_t)-1 && setgroups(ngroups, groups) < 0) {
>>          virReportSystemError(errno, "%s",
>>                               _("cannot set supplemental groups"));
>>
>>
> 
> Yes, there's an overflow:
> 2019-08-21 18:25:37.943+0000: 123145413914624: debug : virSetUIDGID:1046 : setgroups 23 max 16
> 
> Related samba ticket (it also has references to the python and dovecot
> issues):
> https://bugzilla.samba.org/show_bug.cgi?id=8773

I wonder if gnulib could provide a workaround setgroups() that overcomes
this issue (it's better to maintain such a patch there, where it
benefits multiple programs, rather than just in libvirt).

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [libvirt] Fwd: libvirtd failing on MacOS in setgroups

[Bruno Haible]

Hi Eric,

> I wonder if gnulib could provide a workaround setgroups() that overcomes
> this issue

I don't see how a workaround could look like. The problem is not the value
of NGROUPS_MAX in user-space, but the same value NGROUPS_MAX in the kernel.
More precisely, in the Darwin kernel file bsd/kern/kern_prot.c there is a
function 'setgroups1', that contains the common implementation of the
setgroups() and initgroups() system call, and this function fails with EINVAL
if the number of groups in the set is > NGROUPS. In the kernel sources,
NGROUPS is defined as NGROUPS_MAX, and NGROUPS_MAX is defined as 16.

So, the situation on macOS has not changed since this page was written:
https://www.j3e.de/ngroups.html

What kind of workaround are you imagining? That we override open(),
access(), eaccess() to call setgroups() first, in an intelligent way?
That would be quite gross.

For what purpose is libvirt or QEMU using setgroups()?

Bruno

Re: [libvirt] Fwd: libvirtd failing on MacOS in setgroups

[Marcus Furlong]

On Fri, 30 Aug 2019 at 21:33, Bruno Haible <bruno@clisp.org> wrote:
>
> Hi Eric,
>
> > I wonder if gnulib could provide a workaround setgroups() that overcomes
> > this issue
>
> I don't see how a workaround could look like. The problem is not the value
> of NGROUPS_MAX in user-space, but the same value NGROUPS_MAX in the kernel.
> More precisely, in the Darwin kernel file bsd/kern/kern_prot.c there is a
> function 'setgroups1', that contains the common implementation of the
> setgroups() and initgroups() system call, and this function fails with EINVAL
> if the number of groups in the set is > NGROUPS. In the kernel sources,
> NGROUPS is defined as NGROUPS_MAX, and NGROUPS_MAX is defined as 16.
>
> So, the situation on macOS has not changed since this page was written:
> https://www.j3e.de/ngroups.html
>
> What kind of workaround are you imagining? That we override open(),
> access(), eaccess() to call setgroups() first, in an intelligent way?
> That would be quite gross.
>
> For what purpose is libvirt or QEMU using setgroups()?

FWIW I compiled libvirt without the setgroups code on Mac and it
worked as expected. Not sure what the implications of that are though?

Marcus.

-- 
Marcus Furlong

Re: [libvirt] Fwd: libvirtd failing on MacOS in setgroups

[Bruno Haible]

Marcus Furlong wrote:
> FWIW I compiled libvirt without the setgroups code on Mac and it
> worked as expected. Not sure what the implications of that are though?

OK, then the fix would be to not use setgroups on Mac, and nothing to do
in gnulib. Right?

Bruno

Re: [libvirt] Fwd: libvirtd failing on MacOS in setgroups

[Daniel P. Berrangé]

On Sat, Sep 28, 2019 at 01:36:15PM +0200, Bruno Haible wrote:
> Marcus Furlong wrote:
> > FWIW I compiled libvirt without the setgroups code on Mac and it
> > worked as expected. Not sure what the implications of that are though?
> 
> OK, then the fix would be to not use setgroups on Mac, and nothing to do
> in gnulib. Right?

Not calling setgroups means the QEMU process doesn't run with any of
the supplementary groups associated with its user account, so this is
not really a working solution. It re-introduces the bug that the
setgroups call was added to fix.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [libvirt] Fwd: libvirtd failing on MacOS in setgroups

[Bruno Haible]

Daniel P. Berrangé wrote:
> > > FWIW I compiled libvirt without the setgroups code on Mac and it
> > > worked as expected. Not sure what the implications of that are though?
> > 
> > OK, then the fix would be to not use setgroups on Mac, and nothing to do
> > in gnulib. Right?
> 
> Not calling setgroups means the QEMU process doesn't run with any of
> the supplementary groups associated with its user account, so this is
> not really a working solution. It re-introduces the bug that the
> setgroups call was added to fix.

For what purpose is libvirt or QEMU using setgroups()? What goes wrong if
setgroups() fails?

The problem is that the Darwin kernel does not support setting more than
NGROUPS_MAX (= 16) groups. So
  - What happens when you have a user account which is in more than 16
    groups? What do other processes do in this sitation?
  - Is using the first 16 groups and ignoring the extra ones an acceptable
    solution?

Bruno

Re: [libvirt] Fwd: libvirtd failing on MacOS in setgroups

[Daniel P. Berrangé]

On Mon, Sep 30, 2019 at 02:06:07PM +0200, Bruno Haible wrote:
> Daniel P. Berrangé wrote:
> > > > FWIW I compiled libvirt without the setgroups code on Mac and it
> > > > worked as expected. Not sure what the implications of that are though?
> > > 
> > > OK, then the fix would be to not use setgroups on Mac, and nothing to do
> > > in gnulib. Right?
> > 
> > Not calling setgroups means the QEMU process doesn't run with any of
> > the supplementary groups associated with its user account, so this is
> > not really a working solution. It re-introduces the bug that the
> > setgroups call was added to fix.
> 
> For what purpose is libvirt or QEMU using setgroups()? What goes wrong if
> setgroups() fails?

QEMU potentially needs access to files owned by a supplementary group.
On Linux for example, /dev/kvm is often owned by 'kvm' group, but the
'qemu' user on Fedora has 'qemu' group as its primary group. So QEMU
would be unable to open /dev/kvm without the setgroups call to set up
supplementary groups.

> The problem is that the Darwin kernel does not support setting more than
> NGROUPS_MAX (= 16) groups. So
>   - What happens when you have a user account which is in more than 16
>     groups? What do other processes do in this sitation?

Samba appears to use  initgroups on Darwin, while clamping to 16 groups
only:

  https://github.com/samba-team/samba/blob/v4-11-stable/source3/smbd/sec_ctx.c#L248

>   - Is using the first 16 groups and ignoring the extra ones an acceptable
>     solution?

Certainly that's better than just ignoring groups entirely, as it will
work for many more cases, even if not perfect. 

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [libvirt] Fwd: libvirtd failing on MacOS in setgroups

[Bruno Haible]

Daniel P. Berrangé wrote:
> > For what purpose is libvirt or QEMU using setgroups()? What goes wrong if
> > setgroups() fails?
> 
> QEMU potentially needs access to files owned by a supplementary group.
> On Linux for example, /dev/kvm is often owned by 'kvm' group, but the
> 'qemu' user on Fedora has 'qemu' group as its primary group. So QEMU
> would be unable to open /dev/kvm without the setgroups call to set up
> supplementary groups.

Ah, it's libvirt which calls setgroups and qemu which needs the groups.
Then my suggested workaround that consists of overriding setgroups() and
open() won't work.

> >   - Is using the first 16 groups and ignoring the extra ones an acceptable
> >     solution?
> 
> Certainly that's better than just ignoring groups entirely, as it will
> work for many more cases, even if not perfect. 

Hmm. If the group of /dev/kvm comes at 17th group, it will still not work.
I.e. it will be unreliable.

Then, how about if libvirt collects the set of groups that qemu might need
for accessing devices (surely less than 16), then fills up the remaining
up to 16 slots with secondary groups? Admittedly it makes qemu less
self-contained. But given that setgroups() works only for root on macOS [1]
I see no better way.

Bruno

[1] https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/setgroups.2.html

Re: [libvirt] Fwd: libvirtd failing on MacOS in setgroups

[Marcus Furlong]

On Mon, 30 Sep 2019 at 21:05, Bruno Haible <bruno@clisp.org> wrote:
>
> Daniel P. Berrangé wrote:
> > > For what purpose is libvirt or QEMU using setgroups()? What goes wrong if
> > > setgroups() fails?

On macOS, as far as I can see, everything works as expected without it.

So not sure if it's actually needed?

> > QEMU potentially needs access to files owned by a supplementary group.
> > On Linux for example, /dev/kvm is often owned by 'kvm' group, but the
> > 'qemu' user on Fedora has 'qemu' group as its primary group. So QEMU
> > would be unable to open /dev/kvm without the setgroups call to set up
> > supplementary groups.
>
> Ah, it's libvirt which calls setgroups and qemu which needs the groups.
> Then my suggested workaround that consists of overriding setgroups() and
> open() won't work.
>
> > >   - Is using the first 16 groups and ignoring the extra ones an acceptable
> > >     solution?
> >
> > Certainly that's better than just ignoring groups entirely, as it will
> > work for many more cases, even if not perfect.
>
> Hmm. If the group of /dev/kvm comes at 17th group, it will still not work.
> I.e. it will be unreliable.
>
> Then, how about if libvirt collects the set of groups that qemu might need
> for accessing devices (surely less than 16), then fills up the remaining
> up to 16 slots with secondary groups? Admittedly it makes qemu less
> self-contained. But given that setgroups() works only for root on macOS [1]
> I see no better way.

Note that /dev/kvm is for linux and does not exist on macOS.

Unless we identify specific devices on macOS that qemu requires access
to, then something like the following might work?

https://github.com/furlongm/libvirt/commit/01a1d3d0e37c7f81a04da2e9707ac1c39f4642b9

Seems to work correctly for me (virsh capabilities now returns the
correct output, and VMs run).

-- 
Marcus Furlong