[VMC on AWS] Unable to deploy Service Mesh with Error: Operation timedout in state POST_CONFIG_VIX

Products

• VMware HCX
• VMware Cloud on AWS

Issue/Introduction

Symptoms:

• Unable to deploy Service Mesh with Error: Operation timedout in state POST_CONFIG_VIX
• ESXi has 2 management vmk with different network segments.
• ESXi's DNS resolution IP addresses are different for vCenter and HCX.

Cause

When creating Service Mesh, HCX manager tries to connect to ESXi's Management vmk which Fleet appliances will be deployed on.
Before that, HCX manager retrieves the destination information from vCenter.
 
The destination information propagated from vCenter includes the IP address of ESXi that is resolved by DNS server from vCenter.
If HCX manager cannot connect to this IP address, Service Mesh creation will fail with the above error.

Resolution

This is caused by HCX expected behavior mentioned above.
To resolve this, HCX manager has to be able to connect to ESXi's vmk IP address which is ESXi's DNS resolution IP address from vCenter.

For example, If the result of nslookup from vCenter to ESXi is 192.168.0.1 (vmk0), HCX manager has to be able to connect to 192.168.0.1 (vmk0) while Service Mesh creation.