Ports in blocked state due to "opsAgent lldpVim"

Products

• VMware NSX

Issue/Introduction

 

VMs running on ESXi hosts encountered blocked ports.

 

From the ESXi CLI, run the following command to verify a port in this particular link state:

 

Example:

#net-dvs -l | grep -E "port |port.block|volatile.vlan|volatile.status"

port c2####2-c###-4###-8##a-2#########71:

        com.vmware.common.port.block = true , propType = POLICY

        com.vmware.common.port.volatile.vlan = VLAN 0

        com.vmware.common.port.volatile.status = inUse linkUp blocked portID=67108871 Port blocked by admin propType = RUNTIME 

 

Environment

VMware NSX-T Data Center

Cause

Due to "opsAgent lldpVim" that make the thread busy.

From /var/run/log/nsx-syslog.log :

• timeout messages while trying to communicate to MP is seen.

nsx-opsagent[2101247]: NSX 2101247 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="nsxa" tid="49412961" level="ERROR" errorCode="MPA41542"] [MP_AddVnicAttachment] RPC call [lro- c2####2-c###-4###-8##a-2#########71] to NSX management plane timeout

 

• Below logs show readdata was called frequently, which made opsagent very busy:

2022-07-14T06:02:01Z nsx-opsagent[2101247]: NSX 2101247 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="framework" tid="2101267" level="INFO"] [ReadData] fd [72] totalByteRead = 8 err = [ Function not implemented : 38]

2022-07-14T06:02:02Z nsx-opsagent[2101247]: NSX 2101247 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="framework" tid="2101268" level="INFO"] [ReadData] fd [72] totalByteRead = 40 err = [ Function not implemented : 38]

2022-07-14T06:02:02Z nsx-opsagent[2101247]: NSX 2101247 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="framework" tid="2101268" level="INFO"] [ReadData] fd [72] totalByteRead = 8 err = [ Function not implemented : 38]

2022-07-14T06:02:03Z nsx-opsagent[2101247]: NSX 2101247 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="framework" tid="2101269" level="INFO"] [ReadData] fd [72] totalByteRead = 40 err = [ Function not implemented : 38]

2022-07-14T06:02:03Z nsx-opsagent[2101247]: NSX 2101247 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="framework" tid="2101269" level="INFO"] [ReadData] fd [72] totalByteRead = 8 err = [ Function not implemented : 38]

 

• Consequently, the RPC channel between opsagent and nsxproxy can become inactive for a long time. Therefore a keep-alive request is sent to opsagent via RPC. If the corresponding keep-alive response is not received by nsxproxy within 1 minute, the RPC server thinks opsagent is inactive and unregisters its services

 

Resolution

NSX-HOST-OPSAGENT

This is issue is resolved in NSX 3.2.1

Workaround:

Anyone of the below options:

(A). Unblock the port state using below steps:

1. Identify all the ports that are in a Blocked state, by running the below command

net-dvs -l | grep -E "port |port.block|volatile.vlan|volatile.status"

2. Identify the port IDs for all connected interfaces on the ESXi host

net-stats -l

3. Match the output of 2 and 3 to see what ports correspond to which vNic interfaces

4. Identify the N-VDS switch name on the host

esxcfg-vswitch -l

5. Run below command to unblock the ports which are in blocked state

net-dvs -s com.vmware.common.port.block=false <N-VDS-NameFromStep4> -p <LogicalSwitchPor UUIDfromStep1>

Example:

net-dvs -s com.vmware.common.port.block=false N-VDS-hostswitch -p  c2####2-c###-4###-8##a-2#########71

(B). Restart the below agents on the ESXi host:

/etc/init.d/nsx-opsagent restart
/etc/init.d/nsx-proxy restart

 

 

Additional Information

Impact/Risks:

VM connectivity lost when port is in blocked state