Unextending an HCX network failed with the error: "Operation timedout in state WAIT_FOR_POWER_ON"

Products

• VMware HCX

Issue/Introduction

• An attempt to unextend a network gets stuck at 43% before rolling back.
• After the failure occurs, the NE HA group is no longer in a Healthy state.
• The following error is observed in  common/logs/admin/app.log:

<timestamps> UTC [NetworkStretchService_SvcThread-15500, j: #####, s: #####, , TxId: ####################-##########-##########-##########-##############################] INFO  c.v.v.h.n.i.UnstretchNetworkHAJobInt- Running in state VALIDATE_UNSTRETCH_STANDBY

<timestamps>UTC [FailureDetectionService_EventListener, , , TxId: ] INFO  c.v.v.h.f.FailedJobEventsListener- Received a Failed Job, jobType: NetworkStretchJobs workflow type: UnstretchNetworkWorkflowInt

<timestamps> UTC [FailureDetectionService_EventListener, , , TxId: ] INFO  c.v.v.h.f.FailedJobEventsListener- jobAndWorkflowTypesMap :44

<timestamps> UTC [FailureDetectionService_EventListener, , , TxId: ] INFO  c.v.v.h.f.FailedJobEventsListener- Skipping the jobType NetworkStretchJobs and workflow Type UnstretchNetworkWorkflowInt from notifying.

<timestamps> UTC [NetworkStretchService_SvcThread-15500, j: #####, s: #####, , TxId: ####################-##########-##########-##########-##############################] ERROR c.v.v.h.n.i.AbstractJobInt- UnstretchNetworkWorkflowInt failed, errorCode:null. stacktrace:null

<timestamps> UTC [NetworkStretchService_SvcThread-15500, j: #####, s: #####, , TxId: ####################-##########-##########-##########-##############################] ERROR c.v.v.h.n.i.AbstractJobInt- NetworkStretchJobs workflow UnstretchNetworkWorkflowInt failed. Error: Interconnect Service Workflow GenerateAndPostConfig failed. Error: Operation timedout in state WAIT_FOR_POWER_ON

<timestamps> UTC [NetworkStretchService_SvcThread-15500, j: #####, s: #####, , TxId:####################-##########-##########-##########-##############################] ERROR c.v.v.h.n.i.UnstretchNetworkHAJobInt- Validation for network unextend for standby appliance:  ####################-##########-##########-##########-############################## failed. Error: Interconnect Service Workflow GenerateAndPostConfig failed. Error: Operation timedout in state WAIT_FOR_POWER_ON

Environment

VMware HCX

Cause

• The VMware Tools service is not running on one of the HCX Network Extension (NE) appliances.
• The status of the VMware Tools service can be confirmed via the vCenter UI:

Resolution

This is a known issue impacting VMware HCX.

Workaround:

1. Ensure VMtools are running on the NE appliance:
   • Use the following KB to CCLI to the affected HCX appliance.
     • How to check HCX appliances status in CLI.
     • Follow up to step 3-3.
     • Type <SSH> to login to the appliance.
   • Once logged into the appliance, run the following command to check and start VMtools service.
     <systemctl | grep -i tools>
<systemctl start vmtoolsd.service>
2. Recover HA from HCX UI
   • For more information Managing Network Extension High Availability
3. Force unextend the network
   • For more information Removing a Network Extension

Additional Information

Enable Mobility Optimized Networking failed