Remediation a Cluster with vSphere Lifecycle Manager that has NSX-T configure fails with error "Failed to run health checks for NSX-T on 'CLUSTER NAME"

Products

• VMware vSphere ESXi

Issue/Introduction

• You get below error while Remediating the cluster with vLCM that has NSX-T configured cluster : 

• In the vCenter's /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log, we see messages similar to the below:

YYYY-MM-DDTHH:MM:SS.XXXZ info vmware-vum-server[49816] [Originator@6876 sub=EHP] Acquiring SAML token with extension certificate...
YYYY-MM-DDTHH:MM:SS.XXXZ error vmware-vum-server[22005] [Originator@6876 sub=vmomi.soapStub[3051]] Initial service state request failed, disabling pings; /sso-adminserver/sdk/vsphere.local, <last binding: <<TCP '127.0.0.1 : 12345'>, <TCP '127.0.0.1 : 443'>>>, HTTP Status:405 'Method Not Allowed'
YYYY-MM-DDTHH:MM:SS.XXXZ info vmware-vum-server[12257] [Originator@6876 sub=SsoClient] Successfully acquired token: SamlToken [subject={Name: vpxd-extension-#######-########-###########; Domain:vsphere.local}, groups=[{Name: Users; Domain:vsphere.local}, {Name: SolutionUsers; Domain:vsphere.local}, {Name: SystemConfiguration.Administrators; Domain:vsphere.local}, {Name: ComponentManager.Administrators; Domain:vsphere.local}, {Name: LicenseService.Administrators; Domain:vsphere.local}, {Name: ActAsUsers; Domain:vsphere.local}, {Name: ServiceProviderUsers; Domain:vsphere.local}, {Name: vStatsGroup; Domain:vsphere.local}, {Name: Everyone; Domain:vsphere.local}], delegationChain=[], startTime=YYYY-MM-DD HH:MM:SS.MS, expirationTime=YYYY-MM-DD HH:MM:SS.MS, renewable=false, delegable=true, isSolution=true,confirmationType=1]
[..]
YYYY-MM-DDTHH:MM:SS.XXXZ info vmware-vum-server[49816] [Originator@6876 sub=Default] [SessionHandler 96] ExtractSessionInfo: username:#####, domain:#####.com, isAdministrator:false
YYYY-MM-DDTHH:MM:SS.XXXZ info vmware-vum-server[49816] [Originator@6876 sub=VumVapiAuthzFilter] [RequireAdminUserAuthz 332] RequireAdminUserAuthz::Invoke Method is com.vmware.cdc.change_log.get_changes
YYYY-MM-DDTHH:MM:SS.XXXZ info vmware-vum-server[49816] [Originator@6876 sub=VumVapiAuthzFilter] [RequireAdminUserAuthz 336] RequireAdminUserAuthz::Invoke Sevice ID is com.vmware.cdc.change_log
YYYY-MM-DDTHH:MM:SS.XXXZ info vmware-vum-server[49816] [Originator@6876 sub=VumVapiAuthzFilter] [RequireAdminUserAuthz 375] RequireAdminUserAuthz::Invoke Method authorization is true ---> access to this method is restricted, users with Admin permission are allowed to call it.
[..]
YYYY-MM-DDTHH:MM:SS.XXXZ info vmware-vum-server[43335] [Originator@6876 sub=EHP] [domain-c####3] [com.vmware.vcIntegrity.lifecycle.health.internal.external_provider] returned status: NOT_OK
YYYY-MM-DDTHH:MM:SS.XXXZ info vmware-vum-server[43335] [Originator@6876 sub=EHP] Entity [domain-c####3] health status is: NOT_OK
[..]
YYYY-MM-DDTHH:MM:SS.XXXZ info vmware-vum-server[15869] [Originator@6876 sub=RemediateClusterTask] [ApplyHelpers 594] CheckClusterHealth - health check error - (cluster id = domain-c####3) - (cluster name = Cluster Name) - (perspective = 1) - (status = 3)
[..]
YYYY-MM-DDTHH:MM:SS.XXXZ info vmware-vum-server[15869] [Originator@6876 sub=RemediateClusterTask] [Task, 457] Task:com.vmware.vcIntegrity.lifecycle.RemediateClusterTask ID:5#####-####-####-####-############. Not updating the commitId since not all the hosts in the cluster were successfully remediated.

YYYY-MM-DDTHH:MM:SS.XXXZ error vmware-vum-server[15869] [Originator@6876 sub=RemediateClusterTask] [Task, 457] Task:com.vmware.vcIntegrity.lifecycle.RemediateClusterTask ID:5#####-####-####-####-############. Task Failed. Error: Error:
-->    com.vmware.vapi.std.errors.error
--> Messages:
-->    com.vmware.vcIntegrity.lifecycle.TaskError.HealthCheckFailed<Health Check for 'Cluster Name' failed>

 

 

Environment

VMware NSX-T Datacenter
VMware NSX
VMware vCenter Server
VMware vSphere ESXi

Cause

Invoke Method authorization is restricted and require Admin level permissions to be able to call the function. Hence allow the user / service account Admin level permissions in such a situation. 

Resolution

• Assign Administrator level permission to the User/Service account recorded in the log thread. 

Refer to Creating and assigning a role with privileges