HCX Service Mesh deployment fails with "Error: Operation timedout in state POST_CONFIG_VIX"" when static routes are configured via the HCX UI.

Products

• VMware HCX

Issue/Introduction

• You have edited the compute profile.
• You have added Static routes to the Select Guest Network profile for OS Assisted Migration via the HCX UI.
  
• After the change, HCX Service Mesh re-deployment fails with "Error: Operation timedout in state POST_CONFIG_VIX". 
• The following error is observed in the HCX Manager /common/logs/admin/app.log 

  <Time stamp> [InterconnectService_SvcThread-269, IX:########-####-####-####-#############, J:3b7f5573, , TxId: ########-####-####-####-#############] INFO  c.v.v.h.s.i.GenerateAndPostConfiguration- Output of command /opt/vmware/bin/gwcli on VM vm-ID.
  gwcli: loadDpConfig
  Reloading dpconfig (/tmp/newConfig.proto)...
  error reload cgw config: (id:"gwcli_loadconfig [<Time stamp>]" config_update_result:<configResult:1 rollbackResult:0 configError:"/usr/sbin/ip route add ###.###.###.#/## via ###.###.###.#  table main proto static: exit status 2" rollbackError:"" > )
  <Time stamp> [InterconnectService_SvcThread-269, IX:########-####-####-####-#############, J:3b7f5573, , TxId: ########-####-####-####-#############] INFO  c.v.v.h.s.i.GenerateAndPostConfiguration- Successfully executed command /opt/vmware/bin/gwcli on VM vm-ID.
  ...
  <Time stamp>[InterconnectService_SvcThread-269, IX:########-####-####-####-#############, J:3b7f5573, , TxId: ########-####-####-####-#############] WARN  c.v.v.h.s.i.GenerateAndPostConfiguration- Error from cgw: {"configResult":1,"rollbackResult":0,"configError":"/usr/sbin/ip route add ###.###.###.#/## via ###.###.###.#  table main proto static: exit status 2","rollbackError":"","cmdFailed":false}

• The following error is observed in the HCX SRG appliance /var/log/messages 
  

  <134>1 <Time stamp> <SM>-I1 cgw 1594 - - [Info-Tasker] : runCmdShort Success ip route show table main proto static
  <134>1 <Time stamp> <SM>-I1 cgw 1594 - - [Info-Tasker] : Cmd /usr/sbin/ip route add ###.###.###.#/## via ###.###.##.# table main proto static
  <131>1 <Time stamp> <SM>-I1 cgw 1594 - - [Err-Tasker] : Error: exit status 2 Cmd: ip route add ###.###.###.#/## via ###.###.###.# table main proto static
  <131>1 <Time stamp> <SM>-I1 cgw 1594 - - [Err-Tasker] : cmd (/usr/sbin/ip route add ###.###.###.#/## via ###.###.###.# table main proto static) done, error: exit status 2
  <131>1 <Time stamp> <SM>-I1 cgw 1594 - - [Err-configer] : failed </usr/sbin/ip route add ###.###.###.#/## via ###.###.###.# table main proto static: exit status 2><Error: Invalid prefix for given prefix length.
  <131>1 <Time stamp> <SM>-I1 cgw 1594 - - [Err-configer] : ExeRouteConfig fail: /usr/sbin/ip route add ###.###.###.#/## via ###.###.###.# table main proto static: exit status 2
  <131>1 <Time stamp> <SM>-I1 cgw 1594 - - [Err-configer] : failed to update config, err(/usr/sbin/ip route add ###.###.###.#/## via ###.###.###.# table main proto static: exit status 2)
  <131>1 <Time stamp> <SM>-I1 cgw 1594 - - [Err-commander] : Failed to update tunnel config: /usr/sbin/ip route add ###.###.###.#/## via ###.###.###.# table main proto static: exit status 2

• The following error is observed in the HCX Manager /common/logs/admin/app.log 

  <Time stamp> [InterconnectService_SvcThread-285, IX:########-####-####-####-#############, J:3b7f5573, , TxId: ########-####-####-####-#############] ERROR c.v.v.h.s.i.GenerateAndPostConfiguration- Operation timed out in state POST_CONFIG_VIX. stateStartTime: 1759151456392, stateTimeout at: 1759151756392, now: 1759151766169.
  <Time stamp> [InterconnectService_SvcThread-285, IX:########-####-####-####-#############, J:3b7f5573, , TxId: ########-####-####-####-#############] ERROR c.v.v.h.s.i.GenerateAndPostConfiguration- GenerateAndPostConfiguration failed in state POST_CONFIG_VIX. Error: Operation timedout in state POST_CONFIG_VIX.
  java.util.concurrent.TimeoutException: Operation timedout in state POST_CONFIG_VIX

Environment

VMware HCX 

Cause

This is due to incorrect network configuration, such as an incorrect net mask or gateway. 

Resolution

Use the correct network information when configuring the static route.