NSX Manager upgrade pre-checks failure for Host Transport Node with Error Message "Latency profile can not be configured during upgrade"

Products

• VMware NSX

Issue/Introduction

• During NSX Manager upgrade pre-checks, the pre-check for the Host Transport Node failed relating to the presence of latency profiles within the environment. This pre-check is in place to prevent potential network connectivity of issues post upgrade. The pre-check failure message is below:

"Check if latency is configured during upgrade", "latency_check_description",FAILURE, "Latency profile can not be configured during upgrade. Please manually remove the configuration.

• Latency profiles are created in NSX, often by vRNI feature "Enable latency metric collection". Latency profiles can be queried via API: GET /api/v1/latency-profiles
• If these latency profiles are not removed pre-upgrade, post upgrade VMs may lose connectivity after upgrade to NSX 4.2. Virtual Machines residing on or migrated (vMotioned) to an ESXi host upgraded to 4.2.x lose network connectivity.
• vmkernel log on ESXi host (/var/run/log/vmkernel.log) shows loglines matching below when the post upgrade issue has occurred:

2024-09-03T08:43:51.213Z vmkwarning: cpu32:18618342)WARNING: NetDVS: 3027: Failed to write critical property com.vmware.nsx.latency.param on port ########-####-####-####-############, return :Bad parameter.
2024-09-03T08:43:51.213Z vmkernel: cpu32:18618342)Vmxnet3: 17427: Port_Enable failed for port 0x########

Environment

VMware NSX

Cause

The VMware NSX feature to collect latency statistics is configured using the DVS property com.vmware.nsx.latency.param.

In NSX 4.2, the format of the property was adjusted to reduced the memory footprint. However, the change in format makes the property value to be considered invalid in some cases. When the property is invalid, the port remains blocked.

Resolution

This issue is resolved in NSX version 4.2.1.2

Workaround:

• To recover an already affected VM, one of these operations will restore the network connectivity:
  • Reboot the VM
  • Disconnect and reconnect the vNIC
  • vMotion the VM

• To prevent this issue from occurring, before starting the ESXi host upgrade, follow the below manual process:
  1. Disable vRNI latency in vRNI: Uncheck Enable latency metric collection
  2. List latency profiles:
     
     • For policy-based profiles: GET /policy/api/v1/infra/latency-profiles
     • For MP-based profiles: GET /api/v1/latency-profiles
  3. Delete the latency profiles:

     • For policy-based profiles: DELETE /policy/api/v1/infra/latency-profiles/<latency-profile-id>
     • For MP-based profiles: DELETE /api/v1/latency-profiles/<latency-profile-id>
     Deletion of policy-based profiles is required before MP, otherwise the MP based profile may be re-created. If the deletion of the latency profile fails due to protected ownership, you can override it with header X-Allow-Overwrite: true
  4. Proceed with the upgrade.
  5. Post upgrade, you may revert the above changes if necessary by re-enabling latency metric collection in VRNI.

If you are proactively addressing this issue for an upgrade and the number of latency profiles in your environment is high, making manual removal impractical, please consider opening a case with Broadcom Support and refer to this KB article.

Additional Information

This issue gets flagged in upgrade pre-check while upgrading to 4.2.1 onwards: "Latency profile cannot be configured during upgrade"

The prevention steps above should be implemented and vRNI latency metric collection disabled prior to upgrading of ESXi Host Transport Node to ensure no data plane impact. Once upgraded, metric collection can be re-enabled.

This issue will also trigger when Aria Operations for Networks was removed before unconfiguring the collector, resulting in an unused IPFIX Collector in NSX configuration:

If the collector can't be deleted through NSX UI, you can use API: 

DELETE https://<nsx-manager>/policy/api/v1/infra/ipfix-dfw-collector-profiles/<collector-profile-id>

Ref.: DELETE /policy/api/v1/infra/ipfix-dfw-collector-profiles/{ipfix-dfw-collector-profile-id}