Converging from vSphere to VCF via VCF Installer task fails when Joining an existing fleet management appliance
Article ID: 430373
Updated On:
Products
Issue/Introduction
When joining an existing fleet management appliance - i.e. adding another VCF instance, to an already existing fleet manager, the converging task may fail with the following error message.
Retry VCF Operations fleet management deployment workflow Retry VCF Operations fleet management deployment workflow
A problem has occurred on the server. Please retry or contact the service provider and provide the reference token. Reference Token:<TOKEN_ID>
Logs state the following
domainmanager.log
<date && Time> ERROR [vcf_dm,6980a80cac1f3696a261dbbd919c0cf8,e4b8] [c.v.e.s.s.GetVcfOpsMgmtDeploymentStatusAction,dm-exec-20] Failed to poll VCF Operations Management deployment status
<date && Time> ERROR [vcf_dm,6980a80cac1f3696a261dbbd919c0cf8,e4b8] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-20] [<TOKEN_ID>] FAILED_TO_POLL_VCF_OPS_MGMT_DEPLOYMENT_STATUS Failed to poll status for VCF Operations fleet management deployment workflow
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to poll status for VCF Operations fleet management deployment workflow
at com.vmware.evo.sddc.sddcmanager.GetVcfOpsMgmtDeploymentStatusAction.execute(GetVcfOpsMgmtDeploymentStatusAction.java:141)
at com.vmware.evo.sddc.sddcmanager.GetVcfOpsMgmtDeploymentStatusAction.execute(GetVcfOpsMgmtDeploymentStatusAction.java:39)
at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.invoke(FsmActionState.java:66)
at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:161)
at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:147)
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.invokeMethod(ProcessingTaskSubscriber.java:403)
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.processTask(ProcessingTaskSubscriber.java:517)
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.accept(ProcessingTaskSubscriber.java:128)
at jdk.internal.reflect.GeneratedMethodAccessor322.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:85)
at com.google.common.eventbus.Subscriber.lambda$dispatchEvent$0(Subscriber.java:71)
at com.vmware.vcf.common.tracing.TraceRunnable.run(TraceRunnable.java:63)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.springframework.web.client.HttpClientErrorException$NotFound: 404 : "{"errorCode":"TA_TASK_NOT_FOUND","arguments":["<TASK_ID/ TASK_ID/ TASK_ID>"],"message":"Task with ID <TASK_ID/ TASK_ID/ TASK_ID> not found","referenceToken":"<TOKEN_NUMBER>"}"
at org.springframework.web.client.HttpClientErrorException.create(HttpClientErrorException.java:112)
at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:183)
at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:137)
at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63)
at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:942)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:891)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:790)
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:672)
at com.vmware.evo.sddc.sddcmanager.GetVcfOpsMgmtDeploymentStatusAction.execute(GetVcfOpsMgmtDeploymentStatusAction.java:88)
... 16 common frames omitted
Task = 404 not found - In this scenario the VCF Installer is used to deploy a new SDDC manager.
Caused by: org.springframework.web.client.HttpClientErrorException$NotFound: 404 : "{"errorCode":"TA_TASK_NOT_FOUND","arguments":["<TASK_ID/ TASK_ID/ TASK_ID>"],"message":"Task with ID <TASK_ID/ TASK_ID/ TASK_ID> not found","referenceToken":"<TOKEN_NUMBER>"}"
Attempts to export task <TASK_ID/ TASK_ID/ TASK_ID> via https://knowledge.broadcom.com/external/article?articleNumber=314620, on the VCF installer fail.
Environment
vSphere 9.X
Cause
The Task not found, indicates that the task is ONLY on the SDDC side
Caused by: org.springframework.web.client.HttpClientErrorException$NotFound: 404 : "{"errorCode":"TA_TASK_NOT_FOUND","arguments":["<TASK_ID/ TASK_ID/ TASK_ID>"],"message":"Task with ID <TASK_ID/ TASK_ID/ TASK_ID> not found","referenceToken":"<TOKEN_NUMBER>"}"
Attempts to export task via vault fails as per https://knowledge.broadcom.com/external/article?articleNumber=314620 fails
cat workflow.json
{
"arguments" : [],
"errorCode" : "VCF_RUNTIME_ERROR",
"message" : "Something went wrong. Please retry or contact the service provider and provide the reference token.",
"referenceToken" : "BP5MDE"
}Reason behind it is that the workflow involved, is an INTERNAL SDDC workflow - i.e. <TASK_ID/ TASK_ID/ TASK_ID>.
Said workflow, only exists on the SDDC side, even though there is no error present on the SDDC, in the UI.
Resolution
0. Take a snapshot of the SDDC manager
1. Retry the SDDC "internal" Workflow, via SSH
2. This command should provide workflow, as well as failure information# curl localhost/domainmanager/workflows/<TASK_ID/ TASK_ID/ TASK_ID> -X GET
3.The command below, will retry the workflow, and provide the workflow error in the UI # curl localhost/domainmanager/workflows/<TASK_ID/ TASK_ID/ TASK_ID> -X PATCH
4. Process may fail at various subtask, or stage, [i.e the initial failure that is not present in the UI]:
- NTP error i.e. port 123
- Error 504 Gateway port 22/ 443 not opened
etc
Feedback
