Error: On-demand uplink metrics test failed due to appliance partition errors in HCX

Products

• VMware HCX

Issue/Introduction

The HCX Manager UI displays a failure during the "On-demand uplink metrics test" within the Transport Analytics wizard, even when the Service Mesh status and Data-Plane tunnels are reported as Up/Healthy.

Symptoms:

• The error message "On-demand uplink metrics test failed" appears in the HCX Manager UI.
• Clicking the information icon for the appliance reveals partition errors.
• Manual Service Mesh Resync does not clear the error.
• Log signatures from the HCX Manager logs (applianceSummary.txt) health status for the appliance show:

  "reasons": [
      "System state is critical",
      "Config engine is in systemdBad state.",
      "Partition root has problem to create new files. Err: open /fscheck####: read-only file system"
  ]

Environment

• VMware HCX 4.11.x

Cause

This issue occurs when an HCX Interconnect (IX) or Network Extension (NE) appliance enters a read-only file system state. The HCX Manager UplinkUnderlayMetricOnDemandCollectorJob fails because the appliance cannot write diagnostic result files to the local disk.

Resolution

To restore functionality, the affected appliance must be redeployed to reset the file system state.

Note: If there are active extended networks, a maintenance window must be scheduled to perform the redeploy as it will cause a brief disruption to traffic on those networks.

1. Log in to the HCX Connector (or Cloud Manager) UI.
2. Navigate to Infrastructure > Service Mesh.
3. Select View Appliances.
4. Identify the appliance reporting the systemdBad or read-only file system state.
5. Select the appliance and click Redeploy.
6. In the confirmation dialog, select Force Redeploy.
7. Once the redeploy completes and the appliance status returns to "Up", re-run the On-demand uplink metrics test.

For ongoing status updates regarding this behavior, subscribe to this article. Reference: Subscribe to a Broadcom knowledge article by article or product

Additional Information

• To speak with a customer representative or a Support Engineer see Contact Support. Scroll to the bottom of the page and click on your respective region.
• For instructions on gathering logs for further investigation, see Creating and managing Broadcom cases.