Suspend Guest of Linux VM with open-vm-tools may fail

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• A "Suspend Guest" operation on a Linux guest running any version of open-vm-tools and with SELinux enabled may stall and ultimately fail.
• You see the failures such as:
  
  - "Failed to suspend the virtual machine" message display.
  - Nothing happens and the "Suspend Guest" button is reactivated. IPv4 connections may be closed.
  - A delayed suspend happens but the IPv4 addresses are lost when the VM is resumed.
   
• Even an apparent "stall" which exceeds 30 seconds is an indication of the problem.

To confirm the issue:

Check the contents of /var/log/vmware-network.log file.

1. Determine the total time the "network" script took to complete from the time stamps at the beginning and end of the file. If the script took more than 30 seconds to execute, it indicates that the script may be having issues invoking the NetworkManager.
    
2. Further confirmation of the issue is indicated by the presence of messages similar to:
   
   "Error org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken."

Check if SELinux is enabled in "enforcing" mode and if the vmtools_t processes can not access the NetworkManager.

1. Run this command:
   
   sestatus
   
   This command checks if SELinux is enabled and the current mode is "enforcing".
    
2. Run this command:
   
   grep denied /var/log/audit/audit.log | grep vmtools | tail -1
   
   You see a similar similar message confirming the issue similar to:
   
   type=USER_AVC msg=audit(1565910160.495:124): pid=895 uid=81 auid=4294967295 ses=4294967295 subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc:  denied  { send_msg } for msgtype=signal interface=org.freedesktop.NetworkManager.IP6Config member=PropertiesChanged dest=org.freedesktop.DBus spid=1037 tpid=3455 scontext=system_u:system_r:NetworkManager_t:s0 tcontext=system_u:system_r:vmtools_unconfined_t:s0 tclass=dbus permissive=0  exe="/usr/bin/dbus-daemon" sauid=81 hostname=? addr=? terminal=?'UID="dbus" AUID="unset" SAUID="dbus"

Cause

This issue occurs because the Selinux policies in effect are not allowing the /etc/vmware-tools/scripts/vmware/network script to interact with the NetworkManager through the dbus-send command. As a result, the network script which normally would complete in 1 or 2 seconds, exceeds the current 30 second limit for task execution.The "script suspend" or "script resume" times out and current network information may be lost.

Resolution

To resolve this issue, update the selinux-policy and selinux-policy-targeted packages from the Linux vendor. Update these two packages to the latest version available for your Linux release.  If the issue persists, then consider the following workaround.

Note: The issue has been corrected in the following Linux releases and their respective selinux-* packages:

RHEL 7.7 (beta)

• selinux-policy-targeted-3.13.1-246.el7.noarch
• selinux-policy-3.13.1-246.el7.noarch

RHEL 8.1 (beta)

• selinux-policy-targeted-3.14.3-9.el8.noarch
• selinux-policy-3.14.3-9.el8.noarch

Fedora30

• selinux-policy-3.14.3-42.fc30.noarch
• selinux-policy-targeted-3.14.3-42.fc30.noarch

Workaround:
To work around this issue, create an exemption for the vmtools/NetworkManager denied access by using the audit2allow command to generate a local loadable SELinux policy module.

Note: The following commands require root priviledges to be executed.

1. Run this command
   
   grep denied /var/log/audit/audit.log | grep vmtools | tail -1 | \ audit2allow -a -M fix-vmtools-NetworkMgr.
   
   This should generate two files:
   
   fix-vmtools-NetworkMgr.te  - type enforcement file
   fix-vmtools-NetworkMgr.pp  - packaged policy module
   
   The fix-vmtools-NetworkMgr.te contents should be similar to:
   
     module fix-vmtools-NetworkMgr 1.0;
   
     require {
             type vmtools_unconfined_t;
             type NetworkManager_t;
             class dbus send_msg;
     }
   
     #============= NetworkManager_t ==============
     allow NetworkManager_t vmtools_unconfined_t:dbus send_msg;
     
2. Now install the policy module by running this command:
   
   semodule -i fix-vmtools-NetworkMgr.pp
    
3. To remove this local policy module, run this command:
   
   semodule -r fix-vmtools-NetworkMgr