vCenter Server 8.0 Alarm actions which run scripts fail with “Command must exist/be executable”

Products

• VMware vCenter Server

Issue/Introduction

• Security in VCSA services has been increased and most services no longer run with root privileges. 
• One of them is VPXD. Alarm scripts created before vCenter Server 8.0 may no longer work after the upgrade, if it didn't have the correct OS permissions and executed privileged actions. 
• This KB is to notify why after an upgrade the failures are seen in alarm scripts, the cause and how to address this behavior.

Symptoms:
VPXD logs contains the following error messages related to the SchedulerTask launched for the respective triggered alarm action:

In the /var/log/vmware/vpxd/vpxd.log you may see similar entries:
2022-03-10T10:50:51.511Z info vpxd[17494] [Originator@6876 sub=vpxLro opID=ProcessEventNotifications-647d2f21-7eabb9ae] [VpxLRO] -- BEGIN lro-97183 --  -- ScheduledTaskLRO --
2022-03-10T10:50:51.514Z error vpxd[17494] [Originator@6876 sub=SysCommandPosix opID=ProcessEventNotifications-647d2f21-7eabb9ae] Invocation of process: '/root/alarmscript.sh' failed: Command must exist/be executable : /root/alarmscript.sh
2022-03-10T10:50:51.514Z error vpxd[17494] [Originator@6876 sub=SysCommandPosix opID=ProcessEventNotifications-647d2f21-7eabb9ae] Stream (1) is not initialized. Async/Invoke() must be called first.
2022-03-10T10:50:51.514Z verbose vpxd[17494] [Originator@6876 sub=MoScheduledTask opID=ProcessEventNotifications-647d2f21-7eabb9ae] [ScheduledTaskActionLRO] Failed to run script: Failed to invoke command: Stream is not initialized
2022-03-10T10:50:51.514Z verbose vpxd[17494] [Originator@6876 sub=MoEvent opID=ProcessEventNotifications-647d2f21-7eabb9ae] [EventManagerMo] Event[2287]: Object[datacenter-3(Datacenter1)], Type[vim.event.AlarmScriptFailedEvent]
2022-03-10T10:50:51.514Z info vpxd[17494] [Originator@6876 sub=vpxLro opID=ProcessEventNotifications-647d2f21-7eabb9ae] [VpxLRO] -- FINISH lro-97183
2022-03-10T10:50:51.516Z error vpxd[17471] [Originator@6876 sub=MoScheduledTask opID=ProcessEventNotifications-647d2f21-7eabb9ae] Script failed to execute: Command must exist/be executable : /root/alarmscript.sh

vCenter UI shows a vim.event.AlarmScriptFailedEvent error type event under the Monitor tab with message "Alarm <alarm_name> on <managed_object_name> did not complete script"

Environment

VMware vCenter Server 8.0.0

Cause

If the script worked until upgrading to vCenter Server 8.0 and above, the cause of failure is the script does not have the correct OS permissions or contains privileged commands. Since vCenter Server 8.0, alarm scripts are executed by the VCSA Photon OS user "vpxd" which doesn't have root privileges.

Resolution

Alarm scripts must be owned by VCSA Photon OS user "vpxd". OS Permissions should look like this:
root@wdc-10-180-202-111 [ /alarmscripts ]# ls -lah alarmscript.sh
-rwxr--r-- 1 vpxd root 45 Mar 10 10:45 alarmscript.sh

Additionally, if the script continues to fail even with correct OS permissions, you need to inspect the script for privileged actions. We do not recommend alarms scripts to contain actions which require privileges more than the regular OS user "vpxd" has. If for some reason you require alarm scripts to be execute with root privileges, you can temporarily allow this the following way, but only until you redesign these scripts to be executed in a non-root fashion, since continuing to do this reduces VCSA security:
1. Copy your original alarmscript.sh into another file (e.g. root_alarmscript.sh)
2. Edit your original alarmscript.sh to only invoke the command "sudo <absolute_path_to_root_alarmscript> <arguments_to_root_alarmscript>" and make sure that alarmscript.sh has the correct OS permission as indicated above.
3. Append the privileged root_alarmscript.sh to the sudoers file like so:
a. open /etc/sudoers
b. add the line "vpxd ALL= NOPASSWD:  <absolute_path_to_root_alarmscript> <arguments>"
Recommendation is to eliminate root privileged actions from alarms scripts.