vCenter Server 8.0 Alarm actions which run scripts fail with “Command must exist/be executable”

Products

• VMware vCenter Server

Issue/Introduction

In vCenter Server 8.0 and later, security measures have been enhanced and many services no longer run with root privileges such as VPXD. If alarm scripts created prior to upgrading to vCenter Server 8.0 attempt to execute privileged actions without the necessary OS permissions, they may fail after the upgrade.

• vCenter - /var/log/vmware/vpxd/vpxd.log contains the following error messages related to the SchedulerTask launched for the respective triggered alarm action:
  
  YYYY-MM-DDTHH:MM:SS.SSSZ info vpxd[17494] [Originator@6876 sub=vpxLro opID=ProcessEventNotifications-647d2f21-7eabb9ae] [VpxLRO] -- BEGIN lro-97183 --  -- ScheduledTaskLRO --
YYYY-MM-DDTHH:MM:SS.SSSZ error vpxd[17494] [Originator@6876 sub=SysCommandPosix opID=ProcessEventNotifications-647d2f21-7eabb9ae] Invocation of process: '/root/alarmscript.sh' failed: Command must exist/be executable : /root/alarmscript.sh
YYYY-MM-DDTHH:MM:SS.SSSZ error vpxd[17494] [Originator@6876 sub=SysCommandPosix opID=ProcessEventNotifications-647d2f21-7eabb9ae] Stream (1) is not initialized. Async/Invoke() must be called first.
YYYY-MM-DDTHH:MM:SS.SSSZ verbose vpxd[17494] [Originator@6876 sub=MoScheduledTask opID=ProcessEventNotifications-647d2f21-7eabb9ae] [ScheduledTaskActionLRO] Failed to run script: Failed to invoke command: Stream is not initialized
YYYY-MM-DDTHH:MM:SS.SSSZ verbose vpxd[17494] [Originator@6876 sub=MoEvent opID=ProcessEventNotifications-647d2f21-7eabb9ae] [EventManagerMo] Event[2287]: Object[datacenter-3(Datacenter1)], Type[vim.event.AlarmScriptFailedEvent]
YYYY-MM-DDTHH:MM:SS.SSSZ info vpxd[17494] [Originator@6876 sub=vpxLro opID=ProcessEventNotifications-647d2f21-7eabb9ae] [VpxLRO] -- FINISH lro-97183
YYYY-MM-DDTHH:MM:SS.SSSZ error vpxd[17471] [Originator@6876 sub=MoScheduledTask opID=ProcessEventNotifications-647d2f21-7eabb9ae] Script failed to execute: Command must exist/be executable : /root/alarmscript.sh
  
  
• vCenter UI shows a vim event AlarmScriptFailedEvent error type event under the Monitor tab with message "Alarm <alarm_name> on <managed_object_name> did not complete script".

Environment

VMware vCenter Server 8.0 and above

Cause

If the script worked until upgrading to vCenter Server 8.0 and above, the cause of failure is that the script does not have the correct OS permissions or contains privileged commands. Since vCenter Server 8.0, alarm scripts are executed by the VCSA Photon OS user "vpxd", which doesn't have root privileges.

Resolution

Alarm scripts must be owned by the VCSA Photon OS user "vpxd" and group "cis". OS Permissions should look like this:

root@vcsa [ /alarmscripts ]# ls -lah alarmscript.sh
-rwxr--r-- 1 vpxd cis 45 Mar 10 10:45 alarmscript.sh

Additionally, if the script continues to fail even with correct OS permissions, inspect the script for privileged actions. We do not recommend that alarm scripts contain actions that require privileges greater than those of the regular OS user "vpxd".

If, for some reason, it is necessary to execute the alarm scripts with root privileges, this can be temporarily allowed. However, it is only until the script is redesigned that these scripts to be executed in a non-root fashion, since continuing to do this reduces VCSA security:

1. Copy the original alarmscript.sh into another file (e.g., root_alarmscript.sh)
   
   
2. Edit the original alarmscript.sh to only invoke the command
   
   sudo <absolute_path_to_root_alarmscript> <arguments_to_root_alarmscript>

Note: Ensure the alarmscript.sh has the correct OS permission as indicated above.
   
   
3. To eliminate root-privileged actions from alarm scripts, append the privileged root_alarmscript.sh to the sudoers file:
   
   
   1. Edit the /etc/sudoers file:
      
      visudo /etc/sudoers
      
      
   2. Add the line:
      
      vpxd ALL= NOPASSWD: <absolute_path_to_root_alarmscript> <arguments>


   3. Save the file

Additional Information

Should the script still fail to execute, ensure the proper interpreter is set at the top of the script. For example, in BASH use 

#! /bin/bash