NSX Manager upgrade pre-checks failure for Host Transport Node with Error Message "Latency profile can not be configured during upgrade"

Products

VMware NSX

Issue/Introduction

The upgrade pre-check for the ESXi Transport Node fails with the below message relating to the presence of latency profiles within the environment, this pre-check is in place to prevent potential network connectivity of issues post upgrade:
```
"Check if latency is configured during upgrade", "latency_check_description",FAILURE, "Latency profile can not be configured during upgrade. Please manually remove the configuration.
```

Latency profiles are created in NSX, often by vRNI feature "Enable latency metric collection". Latency profiles can be queried via API: GET /api/v1/latency-profiles.
If these latency profiles are not removed pre-upgrade, post upgrade VMs may lose connectivity after upgrade to NSX 4.2. Virtual Machines residing on or migrated (vMotioned) to an ESXi host upgraded to 4.2.x lose network connectivity.

On the ESXi host, entries similar to the below will be encountered in /var/run/log/vmkernel.log:

vmkwarning: cpu32:18618342)WARNING: NetDVS: 3027: Failed to write critical property com.vmware.nsx.latency.param on port ########-####-####-####-############, return :Bad parameter.

vmkernel: cpu32:18618342)Vmxnet3: 17427: Port_Enable failed for port 0x########

Environment

VMware NSX

Cause

The VMware NSX feature to collect latency statistics is configured using the DVS property com.vmware.nsx.latency.param.

In NSX 4.2, the format of the property was adjusted to reduce the memory footprint. However, the change in format makes the property value to be considered invalid in some cases. When the property is invalid, the port remains blocked.

Resolution

This issue is resolved in VMware NSX 4.2.1.2, available at Broadcom downloads.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

Workaround

To recover an already affected VM, one of these operations will restore the network connectivity:
1. Reboot the VM
2. Disconnect and reconnect the vNIC
3. vMotion the VM
To prevent this issue from occurring, before starting the ESXi host upgrade, follow the manual process below:
1. In vRNI, disable latency metric collection: Uncheck Enable latency metric collection
  - Navigate to Settings > Accounts and Datasource, find the NSX Manager Datasource, and turn off the "Data Collection" radio button.
2. In NSX, identify if the latency profile is Policy or MP:
  - For policy-based profiles: GET /policy/api/v1/infra/latency-profiles
    If a profile is returned by this API, NSX has a Policy latency profile configured. Skip to step 3.
  - For MP-based profiles: GET /api/v1/latency-profiles
    If the Policy query returned no result but the MP API returns a profile, NSX has an MP latency profile. Skip to step 4.
3. Deleting a Policy latency profile:
  - DELETE /policy/api/v1/infra/latency-profiles/<"id" identified in step 2>
  - Confirm the Policy profile is removed GET /policy/api/v1/infra/latency-profiles
  - Confirm the Policy profile no longer exists at MP level GET /api/v1/latency-profiles
  - If Policy profile is removed but the profile remains at MP layer, it must be deleted using Step 4 plus the additional header X-Allow-Overwrite: true
4. Deleting an MP latency profile
  - DELETE /api/v1/latency-profiles/<"id" identified in step 2>
5. Proceed with the upgrade.
6. Post upgrade, you may revert the above changes if necessary by re-enabling latency metric collection in VRNI.

Note: If there are multiple profiles, steps 2-4, will need to be iterated until all are removed. If the environment has an excessive number of profiles that cannot be removed manually, please consider opening a case with Broadcom Support and refer to this KB article.

Additional Information

This issue gets flagged in upgrade pre-check while upgrading to 4.2.1 onwards: "Latency profile cannot be configured during upgrade"

The prevention steps above should be implemented and vRNI latency metric collection disabled prior to upgrading of ESXi Host Transport Node to ensure no data plane impact. Once upgraded, metric collection can be re-enabled.

This issue will also trigger when Aria Operations for Networks was removed before unconfiguring the collector, resulting in an unused IPFIX Collector in NSX configuration:

If the collector can't be deleted through NSX UI, you can use API:

DELETE https://<nsx-manager>/policy/api/v1/infra/ipfix-dfw-collector-profiles/<collector-profile-id>

Ref.: DELETE /policy/api/v1/infra/ipfix-dfw-collector-profiles/{ipfix-dfw-collector-profile-id}