During migration from vSphere to VCF, or while deploying a new VCF 9 instance via the VCF Installer, the task fails at the stage of joining an existing Fleet Management Appliance, generating the error: ‘Retry VCF Operations Fleet Management workflow'

Products

• VMware SDDC Manager

Issue/Introduction

• When joining an existing fleet management appliance - i.e. adding another VCF instance, to an already existing fleet manager, the converging task may fail with the following error message.

Retry VCF Operations fleet management deployment workflow          Retry VCF Operations fleet management deployment workflow
A problem has occurred on the server. Please retry or contact the service provider and provide the reference token. Reference Token:<TOKEN_ID>

• From SDDC Manager: /var/log/vmware/vcf/domainmanager/domainmanager.log reports below errors

 

domainmanager.log

<date && Time> ERROR [vcf_dm,6980a80cac1f3696a261dbbd919c0cf8,e4b8] [c.v.e.s.s.GetVcfOpsMgmtDeploymentStatusAction,dm-exec-20]  Failed to poll VCF Operations Management deployment status
<date && Time> ERROR [vcf_dm,6980a80cac1f3696a261dbbd919c0cf8,e4b8] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-20]  [<TOKEN_ID>] FAILED_TO_POLL_VCF_OPS_MGMT_DEPLOYMENT_STATUS Failed to poll status for VCF Operations fleet management deployment workflow
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to poll status for VCF Operations fleet management deployment workflow
        at com.vmware.evo.sddc.sddcmanager.GetVcfOpsMgmtDeploymentStatusAction.execute(GetVcfOpsMgmtDeploymentStatusAction.java:141)
        at com.vmware.evo.sddc.sddcmanager.GetVcfOpsMgmtDeploymentStatusAction.execute(GetVcfOpsMgmtDeploymentStatusAction.java:39)
        at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.invoke(FsmActionState.java:66)
        at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:161)
        at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:147)
        at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.invokeMethod(ProcessingTaskSubscriber.java:403)
        at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.processTask(ProcessingTaskSubscriber.java:517)
        at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.accept(ProcessingTaskSubscriber.java:128)
        at jdk.internal.reflect.GeneratedMethodAccessor322.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:85)
        at com.google.common.eventbus.Subscriber.lambda$dispatchEvent$0(Subscriber.java:71)
        at com.vmware.vcf.common.tracing.TraceRunnable.run(TraceRunnable.java:63)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.springframework.web.client.HttpClientErrorException$NotFound: 404 : "{"errorCode":"TA_TASK_NOT_FOUND","arguments":["<TASK_ID/ TASK_ID/ TASK_ID>"],"message":"Task with ID <TASK_ID/ TASK_ID/ TASK_ID> not found","referenceToken":"<TOKEN_NUMBER>"}"
        at org.springframework.web.client.HttpClientErrorException.create(HttpClientErrorException.java:112)
        at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:183)
        at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:137)
        at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63)
        at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:942)
        at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:891)
        at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:790)
        at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:672)
        at com.vmware.evo.sddc.sddcmanager.GetVcfOpsMgmtDeploymentStatusAction.execute(GetVcfOpsMgmtDeploymentStatusAction.java:88)
        ... 16 common frames omitted 

Task = 404 not found - In this scenario the VCF Installer is used to deploy a new SDDC manager.

Caused by: org.springframework.web.client.HttpClientErrorException$NotFound: 404 : "{"errorCode":"TA_TASK_NOT_FOUND","arguments":["<TASK_ID/ TASK_ID/ TASK_ID>"],"message":"Task with ID <TASK_ID/ TASK_ID/ TASK_ID> not found","referenceToken":"<TOKEN_NUMBER>"}"

Attempts to export task <TASK_ID/ TASK_ID/ TASK_ID> via https://knowledge.broadcom.com/external/article?articleNumber=314620, on the VCF installer fail. 

• This error can also occur during fresh VCF 9 deployment workflow.

Environment

vSphere 9.X

Cause

The Task not found, indicates that the task is ONLY on the SDDC side

Caused by: org.springframework.web.client.HttpClientErrorException$NotFound: 404 : "{"errorCode":"TA_TASK_NOT_FOUND","arguments":["<TASK_ID/ TASK_ID/ TASK_ID>"],"message":"Task with ID <TASK_ID/ TASK_ID/ TASK_ID> not found","referenceToken":"<TOKEN_NUMBER>"}"

Attempts to export task via vault fails as per https://knowledge.broadcom.com/external/article?articleNumber=314620 fails

cat workflow.json
{
   "arguments" : [],
   "errorCode" : "VCF_RUNTIME_ERROR",
   "message" : "Something went wrong. Please retry or contact the service provider and provide the reference token.",
   "referenceToken" : "BP5MDE"
}

Reason behind it is that the workflow involved, is an INTERNAL SDDC workflow - i.e. <TASK_ID/ TASK_ID/ TASK_ID>.
Said workflow, only exists on the SDDC side, even though there is no error present on the SDDC, in the UI.

 

Resolution

0. Take a snapshot of the SDDC manager

1. Retry the SDDC "internal" Workflow, via SSH

2. This command should provide workflow, as well as failure information
# curl localhost/domainmanager/workflows/<TASK_ID/ TASK_ID/ TASK_ID> -X GET

3.The command below, will retry the workflow, and provide the workflow error in the UI
# curl localhost/domainmanager/workflows/<TASK_ID/ TASK_ID/ TASK_ID> -X PATCH

4. Process may fail at various subtask, or stage, [i.e the initial failure that is not present in the UI]:
- NTP error i.e. port 123
- Error 504 Gateway port 22/ 443 not opened
etc