HCX - IX/NE appliance configuration failed. Reason: Failed to upload file to guest
Article ID: 323281
Updated On:
Products
VMware HCX
VMware Cloud on AWS
Issue/Introduction
This resource is to inform about fleet appliance configuration failure instances and how to recover that.
Symptoms:
HCX Interconnect (IX) OR Network Extension (NE) appliances may stuck in CONFIGURED_FAILED stage and below errors/exceptions can be seen in HCX app-engine logs:
Location of App Engine log:
Symptoms:
HCX Interconnect (IX) OR Network Extension (NE) appliances may stuck in CONFIGURED_FAILED stage and below errors/exceptions can be seen in HCX app-engine logs:
2022-07-18 13:26:57.852 UTC [InterconnectService_SvcThread-48803, SM:servicemesh-217560ff-e780-489b-a428-ae5fa89572a3, IX:dbdb1a18-24df-4d38-9df3-aee896303efb, J:a5e5deba, , TxId: 247d23b9-8430-4219-a40d-915d903dc6a3] INFO c.v.v.h.s.i.GenerateAndPostConfiguration- GenerateAndPostConfig Running in state: POST_CONFIG_VIX 2022-07-18 13:26:58.199 UTC [InterconnectService_SvcThread-48803, SM:servicemesh-217560ff-e780-489b-a428-ae5fa89572a3, IX:dbdb1a18-24df-4d38-9df3-aee896303efb, J:a5e5deba, , TxId: 247d23b9-8430-4219-a40d-915d903dc6a3] INFO c.v.v.h.s.i.GenerateAndPostConfiguration- About to push config file /common/logs/admin/GW_CONFIG_X.X.X.X-1658150818081.proto to appliance vm-XXXXXX 2022-07-18 13:26:58.399 UTC [InterconnectService_SvcThread-48803, SM:servicemesh-217560ff-e780-489b-a428-ae5fa89572a3, IX:dbdb1a18-24df-4d38-9df3-aee896303efb, J:a5e5deba, , TxId: 247d23b9-8430-4219-a40d-915d903dc6a3] ERROR c.v.v.h.s.i.GenerateAndPostConfiguration- Update config on cloud gateway failed: File Upload is unsuccessful java.lang.Exception: File Upload is unsuccessful at com.vmware.vchs.hybridity.adapters.vcenter.GuestOperationsFacet.uploadData(GuestOperationsFacet.java:527) at com.vmware.vchs.hybridity.adapters.vcenter.GuestOperationsFacet.uploadFile(GuestOperationsFacet.java:502) at com.vmware.vchs.hybridity.service.interconnect.GenerateAndPostConfiguration.publishConfigViaVIX(GenerateAndPostConfiguration.java:516) at com.vmware.vchs.hybridity.service.interconnect.GenerateAndPostConfiguration.doPostConfigVIX(GenerateAndPostConfiguration.java:390) at com.vmware.vchs.hybridity.service.interconnect.GenerateAndPostConfiguration.handleState(GenerateAndPostConfiguration.java:172) at com.vmware.vchs.hybridity.service.interconnect.AbstractInterconnectJob.run(AbstractInterconnectJob.java:207) at com.vmware.vchs.hybridity.messaging.LoggingJobWrapper.run(LoggingJobWrapper.java:41) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
Location of App Engine log:
- HCX Manager : /common/log/admin/app.log
Cause
HCX Connector/Cloud manager pushes configuration to Interconnect (IX) appliance & Network Extension (NE) appliance during all jobs including migration/stretch, redeployment, update & SM resync.
In some cases, the config download process may get interrupted due to potential connectivity issues in the OnPrem/cloud infra. As a result, IX/NE appliances may stuck in an unhealthy state and eventually all subsequent operations will be impacted including Migration/Network Extension or Upgrade/Redeployment due to missing config state.
In some cases, the config download process may get interrupted due to potential connectivity issues in the OnPrem/cloud infra. As a result, IX/NE appliances may stuck in an unhealthy state and eventually all subsequent operations will be impacted including Migration/Network Extension or Upgrade/Redeployment due to missing config state.
Resolution
HCX manager connection with all fleet appliances is fully dependent upon vCenter infrastructure and that cannot be avoided. However, some of the backend workflows are improved in subsequent HCX releases to help recover from such instances more gracefully.
The recommendation is to upgrade HCX manager and its fleet appliances to 4.5.2.
Workaround:
As a potential workaround the recommendation is to execute "force redeploy" of IX/NE appliances to help recover the configuration related issues.
In case, force redeploy didn't help then please open a service request with VMware support services for further assistance.
The recommendation is to upgrade HCX manager and its fleet appliances to 4.5.2.
Workaround:
As a potential workaround the recommendation is to execute "force redeploy" of IX/NE appliances to help recover the configuration related issues.
In case, force redeploy didn't help then please open a service request with VMware support services for further assistance.
Additional Information
Refer HCX - IX/NE appliance error InvalidGuestLogin: Failed to authenticate with the guest operating system using the supplied credentials
Impact/Risks:
Impact/Risks:
- The impact is applicable to all FLEET appliances.
- Upgrade/Redeploy/Resync workflow related to IX/NE appliances won't be serviced.
- Migration workflow will also be affected in the given state of IX appliance.
- New Network Extension services won't be serviced.
- Existing Network Extension datapath will remain unaffected.
Feedback
Yes
No
Powered by
