NSX Manager upgrade pre-checks failure for Host Transport Node with Error Message "Latency profile can not be configured during upgrade"

Products

• VMware NSX

Issue/Introduction

• During NSX Manager upgrade pre-checks, the pre-check for the Host Transport Node failed relating to the presence of latency profiles within the environment 
  • "Check if latency is configured during upgrade","latency_check_description",FAILURE,"Latency profile can not be configured during upgrade. Please manually remove the configuration
• VMs may lose connectivity after Host Transport Nodes upgrade to NSX 4.2 if Latency Profile is configured for Hosts
  • ESXi Transport Nodes have recently been upgraded to VMware NSX 4.2.
  • Virtual Machines residing on or migrated (vmotioned) to an ESXi host upgraded to 4.2.x loses network connectivity.
• Latency profiles are created in NSX, often by vRNI feature "Enable latency metric collection".
  • Latency profiles can be queried via API: GET /api/v1/latency-profiles
• vmkernel logs on ESXi hosts are similar to (/var/run/log/vmkernel.log):

2024-09-03T08:43:51.213Z vmkwarning: cpu32:18618342)WARNING: NetDVS: 3027: Failed to write critical property com.vmware.nsx.latency.param on port ########-####-####-####-############, return :Bad parameter.
2024-09-03T08:43:51.213Z vmkernel: cpu32:18618342)Vmxnet3: 17427: Port_Enable failed for port 0x########

Environment

VMware NSX 4.2

Cause

The VMware NSX feature to collect latency statistics is configured using the DVS property com.vmware.nsx.latency.param. In NSX 4.2, the format of the property was adjusted to reduced the memory footprint. However, the change in format makes the property value considered invalid in some cases. When the property is invalid, the port remains blocked.

Resolution

This is a known issue impacting VMware NSX.

Workaround:

• To recover an already affected VM, one of these operations will restore the network connection:
  • Reboot the VM,
  • or disconnect and reconnect the vNIC,
  • or vMotion the VM.
• To prevent this issue from happening before starting the ESXi host upgrade follow the below manual process:
  1. List latency profiles: GET /api/v1/latency-profiles 
  2. Disable vRNI latency in vRNI: Uncheck Enable latency metric collection
  3. Delete the latency profiles: DELETE /api/v1/latency-profiles/<latency-profile-id>
     If the deletion of the latency profile fails due to protected ownership, you can override it with header X-Allow-Overwrite: true
  4. Proceed with the upgrade.
  5. Post upgrade, you may revert the above changes if necessary by re-enabling latency metric collection in VRNI.

If you are proactively addressing this issue for an upgrade and the number of latency profiles in your environment is high, making manual removal impractical, please consider opening a case with Broadcom Support and refer to this KB article.

 

Additional Information

This issue may have been flagged by an upgrade pre-check. The prevention steps above should be implemented and VRNI latency metric collection disabled prior to upgrading of ESXi Host Transport Node to ensure no dataplane impact, once upgraded metric collection can be re-enabled.