Failed to Obtain Session to vCenter Server when Retrying (Create/Remove Cluster, Create/Remove Domain) Workflow

Products

• VMware Cloud Foundation

Issue/Introduction

Failing to obtain a session to vCenter Server when retying a failed Day-N workflow (Create/Remove Cluster, Create/Remove Domain).

When trying to log into the vCenter Server manually, everything works as expected. 


/var/log/vmware/vcf/domainmanager/domainmanager.log

Renew VAPI session - failed to obtain new session org.springframework.web.client.ResourceAccessException: I/O error on > POST request for "https://########.###.###/rest/com/vmware/cis/session": 
{"type":"com.vmware.vapi.std.errors.unauthenticated","value":{"error_type":"UNAUTHENTICATED","challenge":"SIGN 
realm=\"##:##:##:##:##:##:##:##:##:##:##\",sts=\"https://########.###.###/sts/STSService/v> sphere.local\", Basic realm=\"vCenter\"","messages":[
]}} at org.springframework.web.client.RestTemplate.createResourceAccessException(RestTemplate.java:890)

Environment

VCF 5.x

Cause

The reason for the failure is that while the cluster/domain was in "ERROR" state, the passwords of the vCenter were rotated. However, those are already "cached" in the execution context, so when the customer tries to start the workflow again, it fails to connect to the vCenter server as it is using old credentials. 

Resolution

The rotated password should be updated in the execution vault. The following KB should be followed to update the vCenter passwords. Re-try an existing workflow by modifying the workflow spec file