HCX Prevented from Deploying Due to Network Profile Prefix Length Configuration Failure

Products

VMware HCX

Issue/Introduction

Deploying the HCX IX Appliance is essential for establishing robust hybrid cloud environments. However, a common configuration mistake—setting the Network Profile prefix length to 0—can cause the deployment to fail. This misconfiguration disrupts critical network routes, leading to the appliance entering a disconnected state and rendering HCX migration services inoperative.

When the prefix length is misconfigured (e.g., set to 0), the HCX IX appliance deployment fails, and the following log messages appear in app.log:

[InterconnectService_SvcThread-55436, IX:#############, J:#############, , TxId: #############] INFO c.v.v.h.s.i.GenerateAndPostConfiguration- GenerateAndPostConfig Running in state POST_CONFIG_VIX for appliance id ############# name HCX-IX-I1 [InterconnectService_SvcThread-55436, IX:#############, J:#############, , TxId: #############] INFO c.v.v.h.s.i.GenerateAndPostConfiguration- Output of command /opt/vmware/bin/gwcli on VM vm-xxxxx. gwcli: loadDpConfig Reloading dpconfig (/tmp/newConfig.proto)... error reload cgw config: (id:"gwcli_loadconfig [2023-06-21 03:24:38.565836131 +0000 UTC m=+0.142075102]" config_update_result:<configResult:1 rollbackResult:1 configError:"/usr/sbin/ip route add default via x.x.x.x table main proto static:exit status 1" rollbackError:"/usr/sbin/ip route add default via x.x.x.x table main proto static:exit status 1" > ) [InterconnectService_SvcThread-55436, IX:#############, J:cfde6185, , TxId: #############] WARN c.v.v.h.s.i.GenerateAndPostConfiguration- Error from cgw: {"configResult":1,"rollbackResult":1,"configError":"\/usr\/sbin\/ip route add default via x.x.x.x table main proto static:exit status 1","rollbackError":"\/usr\/sbin\/ip route add default via x.x.x.x table main proto static:exit status 1","cmdFailed":false} [InterconnectService_SvcThread-55436, IX:#############, J:#############, , TxId: #############] WARN c.v.v.h.s.i.GenerateAndPostConfiguration- Failed to post the config using VIX for the appliance #############. Retrying the operation. java.lang.RuntimeException: Failed applying appliance configuration. Please check Service Mesh configuration. Please retry operation after ServiceMesh resync. at com.vmware.vchs.hybridity.service.interconnect.GenerateAndPostConfiguration.doPostConfigVIX(GenerateAndPostConfiguration.java:421) at com.vmware.vchs.hybridity.service.interconnect.GenerateAndPostConfiguration.handleState(GenerateAndPostConfiguration.java:174) at com.vmware.vchs.hybridity.service.interconnect.AbstractInterconnectJob.run(AbstractInterconnectJob.java:210) at com.vmware.vchs.hybridity.messaging.LoggingJobWrapper.run(LoggingJobWrapper.java:41) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750)

Upon further inspection using the CCLI>LIST command, the appliance appears in a disconnected state

-------------------------------------------------------------------------|

| Node | Id | Address | State | Selected|

|------------------------------------------------------------------------|

| HCX-IX-I1 | 0 | <x.x.x.x>:9443 | Disconnected | |

|------------------------------------------------------------------------|

Environment

VMware HCX

Cause

The deployment failure occurs when users incorrectly set the prefix length within the Network Profile to 0. This misconfiguration affects the default routing configuration on the HCX IX appliance, leading to errors during the configuration push process. Specifically, the system attempts to add a default route with an invalid prefix length, resulting in log errors like:

error reload cgw config: ... "/usr/sbin/ip route add default via x.x.x.x table main proto static:exit status 1"

Additionally, the appliance remains in a disconnected state post-deployment, as evidenced by the CCLI output. The root cause lies in the HCX Network Profile UI allowing users to set a prefix length below 8 bits, which is invalid for most network configurations.

(the misconfiguration is shown below)

Resolution

To resolve the deployment failure caused by an incorrect prefix length

1. Correct the Prefix Length
  - Access the Network Profile
    - Navigate to the Network Profile settings within the HCX UI.
  - Set a Valid Prefix Length:
    - Ensure the prefix length is set to a valid value, typically between 8 and 30 bits, based on your network segmentation requirements.
    - Avoid setting the prefix length to 0, as it is invalid and leads to configuration errors.
2. Implement Validation
  - Future HCX Releases
    - The system will restrict prefix lengths below 8 bits to prevent such misconfigurations.
    - Users will receive error notifications if they attempt to set an invalid prefix length, ensuring configurations adhere to best practices.
3. Verify Deployment
  - Redeploy the HCX IX Appliance
    - After adjusting the prefix length, redeploy the HCX IX appliance.
  - Confirm Appliance Status:
    - Use the CCLI command to verify that the appliance transitions to a connected state, indicating a successful deployment.

Additional Information

Impact/Risks

Service Disruption: Without a successfully deployed HCX IX appliance, HCX migration services become unavailable, hindering migration and hybrid cloud operations.
Network Instability: Misconfigured network profiles can lead to broader network issues, affecting connectivity and overall service reliability.

Problem Description

Issue: Network Profile allows a prefix length of less than 8 bits.
Expected Behavior: The system should restrict prefix lengths to a minimum of 8 bits to ensure valid and reliable network configurations.