"cannot execute: Permission denied", Alarm scripts are not getting executed on vCenter Server 8.0 U3e / 7.0 U3v and above versions

Products

• VMware vCenter Server

Issue/Introduction

• vCenter alarm scripts created before vCenter Server 8.0 U3e / 7.0 U3v may no longer work after upgrading the vCenter Server, if it didn't have the correct OS permissions and executed privileged actions.

• VPXD logs contains the following error messages related to the SchedulerTask launched for the respective triggered alarm action:

  In the /var/log/vmware/vpxd/vpxd.log you may see similar entries:
  
  /bin/sudo_command_wrapper.sh: line <line_number>: <script_path>: Permission denied
/bin/sudo_command_wrapper.sh: line <line_number>: exec: <script_path>: cannot execute: Permission denied

• vCenter UI shows a vim.event.AlarmScriptFailedEvent error type event under the Monitor tab with message "Alarm <alarm_name> on <managed_object_name> did not complete script"
  
  Sample event:
  Date Time:
 MM/DD/YYYY, HH:MM:SS AM/PM
Type:
 Error
Target:
 <VMName>
Description:
Alarm 'AlarmName' on <VMName> did not complete script:
Event Type Description:
The vCenter Server logs this event if an error occurs while running a script after an alarm triggers.
Possible Causes:
There was an error running the script Action: Fix the script or failure condition
Related events:
There are no related events.

Environment

vCenter Server 8.0 U3e and above versions

vCenter Server 7.0 U3v and above versions

Cause

• The cause of the failure is the script does not have the correct OS permissions or contains privileged commands. Since vCenter Server 8.0 U3e & 7.0 U3v, alarm scripts are executed by dedicated service user "alarms_script" which doesn't have vpxd privileges.
• There have been continuous security enhancements in VCSA services and most services no longer run with root/vpxd privileges. 

Resolution

Change the owner of the script to user 'alarms_script' by following below steps :

• Login to VCSA using SSH client
• Change the file ownership using below command
  
  chown alarms_script:root <script file path>
  
  Example:
  chown alarms_script:root /tmp/alarmtest.sh
  
  Sample result after ownership change :
  root@<hostname> [ /tmp ]# ls -lrt | grep alarm
-rwx------ 1 alarms_script   root                 62 Apr  3 08:59 alarmtest.sh

Additional Information

1. If the script continues to fail even with the correct OS permissions, the script needs to be inspected for privileged actions. It is not recommended to have alarms scripts with the actions which require privileges more than the regular OS user "alarms_script" has. If for some reason the alarm scripts need to be executed with vpxd or higher privileges, it can be temporarily allowed by the following way, but only until these scripts are redesigned to be executed in a non-vpxd fashion, since continuing to do this will not implement the security enhancements done in the release.
   • Copy your original script alarmtest.sh into another file (e.g. root_alarmtest.sh)
   • Edit your original alarmtest.sh to only invoke the command "sudo <absolute_path_to_root_alarmscript> <arguments_to_root_alarmscript>" and make sure that alarmtest.sh has the correct OS permission as indicated above.
   • Append the privileged root_alarmtest.sh to the sudoers file like so:
     • open /etc/sudoers
     • add the line "alarms_script ALL= NOPASSWD:  <absolute_path_to_root_alarmscript> <arguments>"
       
       Recommendation is to eliminate vpxd or higher privileged actions from alarms scripts.
       
       
2. Please refer to the KB vCenter Server 8.0 Alarm actions which run scripts fail with “Command must exist/be executable” where a similar issue is observed on vCenter Server releases prior to 8.0 U3e.