NSX Edge Upgrade Fails Due to PnicStatusInspectionTask

Products

• VMware NSX

Issue/Introduction

• NSX Edge upgrade fails in the NSX_UPGRADE_STAGE_EDGE_POSTCHECK stage, due to the PnicStatusInspectionTask.
• This failure occurs because at least one of the PNICs of the edge is down.

Environment

• VMware NSX
• Edge Form factors: Bare Metal Edge or VM Edge

Cause

The physical link of the NIC is down. Note that the 'physical link' of a VM edge is the Edge's virtual interface.

Resolution

• The failure indicates that at least one of the Edge datapath's NICs is down. It does not indicate which NIC is down. Thus, there are two steps to resolve this issue:

1. 1. Find the NIC that is down.
   2. Determine the cause for the link failure.

• To determine which of the ports is down, invoke the NSX CLI command 'get physical-ports'. Alternatively, invoke the NSX CLI command 'get physical-port' for all datapath interfaces, 'fp-eth0', fp-eth1', etc. The first command gives a list with all the interfaces, in the same format as the second command.

Sample output for 'get physical-port on fp-eth0' interface:

For the down NIC(s), the line for 'LINK' will show a value of 'down'.

• How to determine the cause for the link failure will depending on the form factor:

1. 1. On a Bare Metal Edge:
      • Verify the physical cable connection.
      • Verify that the firmware version of the NIC is compatible with the new version of NSX. After an upgrade of the NSX software, a firmware update may be required. The current firmware version can also be seen in the output above.
      • Verify the admin status of the link in the switch that the Edge interface is connected to.
   2. For VM Edge:
      • On the vCenter UI, make sure that all the network adaptors of this Edge VM are connected to a Virtual Switch.

• On both Bare Metal Edge and VM Edge, if the issue is due an unused interface, the following steps can be used to remove the interface from Dataplane. This step requires Edge reboot:

1. 1. Run the following commands to get the PCI IDs of the interfaces: 
      1. get dataplane device list

Note: This would return the output in the correct PCI format, but, however, it wouldn't match to the interface.
         
         Sample Output:
         
         0000:04:00.0 - VMXNET3 Ethernet Controller  | Vendor: VMware
0000:0b:00.0 - VMXNET3 Ethernet Controller  | Vendor: VMware
0000:13:00.0 - VMXNET3 Ethernet Controller  | Vendor: VMware
0000:1b:00.0 - VMXNET3 Ethernet Controller  | Vendor: VMware
         
         
      2. get interface <interface name>
         
         Note: This would return the output with wrong PCI format, but can match to the interface.
         
         Sample Output:
         
         get interface fp-eth3
Mon Nov 11 2024 UTC 12:16:39.370
Interface: fp-eth3
ID: 3
Link status: up
MAC address: 00:50:56:##:##:##
MTU: 1500
PCI: 0000:04:00:00
         
         
   2. Set the device list. Include only the list of PCI IDs of the interfaces to be used by Dataplane. 
      
      The set command expects the PCI ID in a specific format. 
      Use the PCI ID format obtained in Step #1, part #a: set dataplane device list <list of PCI ID of all used interfaces>.
      
      For example in the Step #1, part #a, where the PCI IDs that were received are:
      
      0000:04:00.0, 0000:0b:00.0, 0000:13:00.0, 0000:1b:00.0
      
      The command that needs to be run to add the interfaces to the device list is: 
      
      set dataplane device list 0000:04:00.0,0000:0b:00.0,0000:13:00.0,0000:1b:00.0
      
      
   3. Reboot the edge node.
      
      Note: The removed interface can be added back to Dataplane, by adding the interface's PCI ID to the device list using the command specified in Step #2
      
      
   4. Changing device list updates the Fast path interface (Fp-ethX) name to MAC address mapping. Update Uplink interface in the Edge Transport Node’s Uplink Profile after the Edge reboots. The new Fast path interface name can be obtained by matching the MAC address to interface name using the ‘get interfaces’ cli output.
      
      Note: The removed interface can be added back to Dataplane, by adding the interface's PCI ID to the device list using the command specified in Step #2. This has to be followed by updating the Uplink in Transport Node uplink profile.