This resource is to inform about fleet appliance configuration failure instances and how to recover that.
Symptoms:
HCX Interconnect (IX) OR Network Extension (NE) appliances may stuck in CONFIGURED_FAILED stage and below errors/exceptions can be seen in HCX app-engine logs:
2022-07-18 13:26:57.852 UTC [InterconnectService_SvcThread-48803, SM:servicemesh-<service mesh ID>, IX:<IX ID>, J:a5e5deba, , TxId: <TX ID>] INFO c.v.v.h.s.i.GenerateAndPostConfiguration- GenerateAndPostConfig Running in state: POST_CONFIG_VIX 2022-07-18 13:26:58.199 UTC [InterconnectService_SvcThread-48803, SM:servicemesh-<service mesh ID>, IX:<IX ID>, J:a5e5deba, , TxId: <TX ID>] INFO c.v.v.h.s.i.GenerateAndPostConfiguration- About to push config file /common/logs/admin/GW_CONFIG_X.X.X.X-1658150818081.proto to appliance vm-XXXXXX 2022-07-18 13:26:58.399 UTC [InterconnectService_SvcThread-48803, SM:servicemesh-<service mesh ID>, IX:<IX ID>, J:a5e5deba, , TxId: <TX ID>] ERROR c.v.v.h.s.i.GenerateAndPostConfiguration- Update config on cloud gateway failed: File Upload is unsuccessful java.lang.Exception: File Upload is unsuccessful at com.vmware.vchs.hybridity.adapters.vcenter.GuestOperationsFacet.uploadData(GuestOperationsFacet.java:527) at com.vmware.vchs.hybridity.adapters.vcenter.GuestOperationsFacet.uploadFile(GuestOperationsFacet.java:502) at com.vmware.vchs.hybridity.service.interconnect.GenerateAndPostConfiguration.publishConfigViaVIX(GenerateAndPostConfiguration.java:516) at com.vmware.vchs.hybridity.service.interconnect.GenerateAndPostConfiguration.doPostConfigVIX(GenerateAndPostConfiguration.java:390) at com.vmware.vchs.hybridity.service.interconnect.GenerateAndPostConfiguration.handleState(GenerateAndPostConfiguration.java:172) at com.vmware.vchs.hybridity.service.interconnect.AbstractInterconnectJob.run(AbstractInterconnectJob.java:207) at com.vmware.vchs.hybridity.messaging.LoggingJobWrapper.run(LoggingJobWrapper.java:41) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
Location of App Engine log: