"Host profile apply failed with error: Error: Access to perform the operation was denied..." Getting the following error when updating the Root or a Service Account password of ESXi host using Host Profile.

Products

• VMware vSphere ESXi
• VMware vCenter Server 8.0

Issue/Introduction

• Using host profile to update the root or service account password gives the following error message , "Host profile apply failed with error: Error: Access to perform the operation was denied...". 
  
  
• Once the host profile application has failed there is no way to clear the compliance check even though all user accounts are in compliance on the host. If you select to only remediate the svc account you will see an error in the logs.

In the vCenter Server log you may see an entry similar to below.

/var/log/vmware/vpxd/vpxd.log -

 

YYYY-MM-DDTHH:MM:SS.041Z error vpxd[05621] [Originator@6876 sub=moHostProfileMgr opID=m9k0c582-28379-auto-lx7-h5:70002995-ab-01] [HostProfileApplyWorker]: Error when Apply host config: ###.###.###, Fault cause: vmodl.fault.SecurityError
-->
YYYY-MM-DDTHH:MM:SS info vpxd[05621] [Originator@6876 sub=moHostProfileMgr opID=m9k0c582-28379-auto-lx7-h5:70002995-ab-01] [ApplyHostProfileToHost]: Remediate finished: ###.###.###
YYYY-MM-DDTHH:MM:SS info vpxd[05621] [Originator@6876 sub=vpxLro opID=m9k0c582-28379-auto-lx7-h5:70002995-ab-01] [VpxLRO] -- FINISH lro-6467736
YYYY-MM-DDTHH:MM:SS info vpxd[06012] [Originator@6876 sub=vpxLro opID=m9k0c582-28379-auto-lx7-h5:70002995-ab] [VpxLRO] -- FINISH task-142904
YYYY-MM-DDTHH:MM:SS error vpxd[06012] [Originator@6876 sub=Default opID=m9k0c582-28379-auto-lx7-h5:70002995-ab] [VpxLRO] -- ERROR task-142904 -- 5263ed0c-cd3c-6974-12b2-c18d0f3adff9(52fcb692-0ea5-ec9b-f2c9-33b47f86b89a) -- HostProfileManager -- vim.profile.host.ProfileManager.applyEntitiesConfiguration: :vim.fault.HostConfigFailed
--> Result:
--> (vim.fault.HostConfigFailed) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    failure = (vmodl.MethodFault) [
-->       (vmodl.fault.SystemError) {
-->          faultCause = (vmodl.MethodFault) null,
-->          faultMessage = <unset>,
-->          reason = "Batch host remediation failed."
-->          msg = ""
-->       }
-->    ]
-->    msg = ""
--> }
--> Args:
-->

->       },
-->       inapplicablePath = <unset>,
-->       requireInput = <unset>,
-->       error = <unset>,
-->       host = 'vim.HostSystem:0f7964bd-70b9-4d40-b925-1877a4a8625a:host-####',
-->       taskListRequirement = <unset>,
-->       taskDescription = (vmodl.LocalizableMessage) [
-->          (vmodl.LocalizableMessage) {
-->             key = "com.vmware.vim.profile.Profile.security.UserAccountProfile.UserAccountProfile.GenerateTaskList.modifyUser.label",
-->             arg = (vmodl.KeyAnyValue) [
-->                (vmodl.KeyAnyValue) {
-->                   key = "name",
-->                   value = "svc-test"
-->                }
-->             ],
-->             message = "'Modifying local user account svc-test'"
-->          },
-->          (vmodl.LocalizableMessage) {
-->             key = "com.vmware.vim.profile.Profile.security.UserAccountProfile.UserAccountProfile.GenerateTaskList.delUser.label",
-->             arg = (vmodl.KeyAnyValue) [
-->                (vmodl.KeyAnyValue) {
-->                   key = "name",
-->                   value = "root"
-->                }
-->             ],
-->             message = "'Removing local user account root'"
-->          }
-->       ],
-->       rebootStateless = <unset>,
-->       rebootHost = true,
-->       faultData = (vmodl.MethodFault) null
-->    }
--> ]

 

Following entries are seen under ESXi host syslog : 

/var/run/log/syslog.log - 

 

YYYY-MM-DDTHH:MM:SSZ In(14) hostprofile[9375643] opID=m9k0c582-28379-auto-lx7-h5:70002995-ab-01-91-613a: ApplyHostConfig called for host profile version 8.0.3
YYYY-MM-DDTHH:MM:SSZ In(14) hostprofile[9375643] opID=m9k0c582-28379-auto-lx7-h5:70002995-ab-01-91-613a: Applying config first....***
YYYY-MM-DDTHH:MM:SSZ In(14) hostprofile[9375643] opID=m9k0c582-28379-auto-lx7-h5:70002995-ab-01-91-613a: Not using on-demand data gathering for profile security_SecurityProfile_SecurityConfigProfile
YYYY-MM-DDTHH:MM:SSZ In(14) hostprofile[9375643] opID=m9k0c582-28379-auto-lx7-h5:70002995-ab-01-91-613a: Calling GatherData() for profile type SecurityConfigProfile
YYYY-MM-DDTHH:MM:SSZ In(14) hostprofile[9375643] opID=m9k0c582-28379-auto-lx7-h5:70002995-ab-01-91-613a: Calling OnRemediateComplete for profile RoleProfile
YYYY-MM-DDTHH:MM:SSZ In(14) hostprofile[9375643] opID=m9k0c582-28379-auto-lx7-h5:70002995-ab-01-91-613a: Not using on-demand data gathering for profile security_UserAccountProfile_UserAccountProfile
YYYY-MM-DDTHH:MM:SSZ In(14) hostprofile[9375643] opID=m9k0c582-28379-auto-lx7-h5:70002995-ab-01-91-613a: Calling GatherData() for profile type UserAccountProfile
YYYY-MM-DDTHH:MM:SSZ In(14) hostprofile[9375643] opID=m9k0c582-28379-auto-lx7-h5:70002995-ab-01-91-613a: Calling RemediateConfig for profile UserAccountProfile
YYYY-MM-DDTHH:MM:SSZ Er(11) hostprofile[9375643] opID=m9k0c582-28379-auto-lx7-h5:70002995-ab-01-91-613a: Exception while applying host config. Exception: (vmodl.fault.SecurityError) {
YYYY-MM-DDTHH:MM:SSZ Er(11)[+] hostprofile[9375643]    dynamicType = <unset>,
YYYY-MM-DDTHH:MM:SSZ Er(11)[+] hostprofile[9375643]    dynamicProperty = (vmodl.DynamicProperty) [],
YYYY-MM-DDTHH:MM:SSZ Er(11)[+] hostprofile[9375643]    msg = 'Access to perform the operation was denied.',
YYYY-MM-DDTHH:MM:SSZ Er(11)[+] hostprofile[9375643]    faultCause = <unset>,
YYYY-MM-DDTHH:MM:SSZ Er(11)[+] hostprofile[9375643]    faultMessage = (vmodl.LocalizableMessage) []
YYYY-MM-DDTHH:MM:SSZ Er(11)[+] hostprofile[9375643] }
YYYY-MM-DDTHH:MM:SSZ Wa(12) hostprofile[9375643] opID=m9k0c582-28379-auto-lx7-h5:70002995-ab-01-91-613a: Exception while applying host config. Backtrace:
YYYY-MM-DDTHH:MM:SSZ Wa(12)[+] hostprofile[9375643]
YYYY-MM-DDTHH:MM:SSZ Wa(12) hostprofile[9375643] opID=m9k0c582-28379-auto-lx7-h5:70002995-ab-01-91-613a:   File "/lib64/python3.11/site-packages/hostprofiles/pyEngine/hostprofilemanager.py", line 1820, in _ApplyHostConfig
YYYY-MM-DDTHH:MM:SSZ Wa(12)[+] hostprofile[9375643]
YYYY-MM-DDTHH:MM:SSZ Wa(12) hostprofile[9375643] opID=m9k0c582-28379-auto-lx7-h5:70002995-ab-01-91-613a:   File "/lib64/python3.11/site-packages/hostprofiles/pyEngine/applyConfigSpec.py", line 4783, in ApplyHostConfig
YYYY-MM-DDTHH:MM:SSZ Wa(12)[+] hostprofile[9375643]
YYYY-MM-DDTHH:MM:SSZ Wa(12) hostprofile[9375643] opID=m9k0c582-28379-auto-lx7-h5:70002995-ab-01-91-613a:   File "/lib64/python3.11/site-packages/hostprofiles/pyEngine/applyConfigSpec.py", line 4368, in ApplyGenericConfig
YYYY-MM-DDTHH:MM:SSZ Wa(12)[+] hostprofile[9375643]
YYYY-MM-DDTHH:MM:SSZ Wa(12) hostprofile[9375643] opID=m9k0c582-28379-auto-lx7-h5:70002995-ab-01-91-613a:   File "/lib64/python3.11/site-packages/hostprofiles/pyEngine/genericProfileBridge.py", line 934, in RecurseRemediateConfig
YYYY-MM-DDTHH:MM:SSZ Wa(12)[+] hostprofile[9375643]
YYYY-MM-DDTHH:MM:SSZ Wa(12) hostprofile[9375643] opID=m9k0c582-28379-auto-lx7-h5:70002995-ab-01-91-613a:   File "/lib64/python3.11/site-packages/hostprofiles/pyEngine/genericProfileBridge.py", line 928, in RecurseRemediateConfig
YYYY-MM-DDTHH:MM:SSZ Wa(12)[+] hostprofile[9375643]
YYYY-MM-DDTHH:MM:SSZ Wa(12) hostprofile[9375643] opID=m9k0c582-28379-auto-lx7-h5:70002995-ab-01-91-613a:   File "/usr/lib/hostprofiles/plugins/security/UserAccountProfile.py", line 646, in RemediateConfig
YYYY-MM-DDTHH:MM:SSZ Wa(12)[+] hostprofile[9375643]     RemoveUser(taskObj)
YYYY-MM-DDTHH:MM:SSZ Wa(12)[+] hostprofile[9375643]
YYYY-MM-DDTHH:MM:SSZ Wa(12) hostprofile[9375643] opID=m9k0c582-28379-auto-lx7-h5:70002995-ab-01-91-613a:   File "/lib64/python3.11/site-packages/pyVim/account.py", line 66, in RemoveUser
YYYY-MM-DDTHH:MM:SSZ Wa(12)[+] hostprofile[9375643]
YYYY-MM-DDTHH:MM:SSZ Wa(12) hostprofile[9375643] opID=m9k0c582-28379-auto-lx7-h5:70002995-ab-01-91-613a:   File "/lib64/python3.11/site-packages/pyVmomi/VmomiSupport.py", line 618, in <lambda>
YYYY-MM-DDTHH:MM:SSZ Wa(12)[+] hostprofile[9375643]
YYYY-MM-DDTHH:MM:SSZ Wa(12) hostprofile[9375643] opID=m9k0c582-28379-auto-lx7-h5:70002995-ab-01-91-613a:   File "/lib64/python3.11/site-packages/pyVmomi/VmomiSupport.py", line 391, in _InvokeMethod
YYYY-MM-DDTHH:MM:SSZ Wa(12)[+] hostprofile[9375643]
YYYY-MM-DDTHH:MM:SSZ Wa(12) hostprofile[9375643] opID=m9k0c582-28379-auto-lx7-h5:70002995-ab-01-91-613a:   File "/lib64/python3.11/site-packages/pyVmomi/SoapAdapter.py", line 1607, in InvokeMethod
YYYY-MM-DDTHH:MM:SSZ Wa(12)[+] hostprofile[9375643]
YYYY-MM-DDTHH:MM:SSZ Wa(12) hostprofile[9375643] opID=m9k0c582-28379-auto-lx7-h5:70002995-ab-01-91-613a: excObj is a runtime fault: (vmodl.fault.SecurityError) {

Environment

• VMware vCenter Server 8.0
• VMware vSphere ESXi 8.0

Cause

If only the service account is selected to be remediated then vCenter will attempt to remove the root user account prompting a message that permission is denied to perform the operation.

Per the following documentation, How Do You Deactivate a Host Profile Component or Subprofile

• NOTE:
Sometimes, deselecting the check box might remove the component or component element from the host. This action is displayed in the task list after the pre-check remediation.

 

Resolution

1. Login to the vCenter UI
2. Go to 'Home' > 'Policies and Profiles' > 'Host Profiles'
3. Right click the Host Profile and click 'Edit Host Profile'
4. In the 'Edit Host Profile' wizard uncheck all boxes
5. Under Security and services > Security setting > Security > User configuration Check the box for all user accounts
6. For each user account In the 'Password' drop-down, choose 'Leave Password unchanged for default account' 
7. Right-click the Host Profile to open the 'Actions' menu. Choose 'Attach/Detach Hosts and Clusters...' and then select the host in the wizard.
8. Right-click the new Host Profile to open the 'Actions' menu. Choose ' Edit Host customization..
9. Select the host which stats Customization required, check the configurations and Finish [ no changes need]
10. Click the hamburger button in the upper left of the screen and select inventory.
11. Right-click the ESXi host in the inventory. The select Host Profiles > Remediate...
12. To confirm success, right-click the host and select Host Profiles > Check Host Profile Compliance.

Additional Information

Steps are similar to Reset ESXi Host Root Password with Host Profile but choose 'Leave Password unchanged for default account' instead of  'Fixed password configuration'