After upgrading to ESXi 7.0 U2a, the Communication Channels show Down in the Host Preparation tab for NSX Data Center for vSphere 6.4.x

Products

• VMware NSX
• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• ESXi is at 7.0 U2a
• Under Network and Security > Installation and Upgrade > Host Preparation, the hosts show their Communication Channels as Down
• Running esxcli network ip connection list | grep 1234 shows no connections to the NSX Controllers (or fewer than there should be)
• Running esxcli network ip connection list | head -n2 && esxcli network ip connection list | grep 5671 shows connections to the NSX Manager, with one connection having a large value in the Recv Q field
• The netcpa daemon is having trouble connecting to vsfwd on port 15002:

$ grep rmqClient /var/run/log/netcpa.log
2021-05-05T17:04:00UTC rmqClient Send returned -1 Err No 32, While Sending Len 66
2021-05-05T17:04:00UTC rmqClient Error Sending Data on Idx 19 Status -1005
2021-05-05T17:04:00UTC rmqClient Unable to Send Data for Id 19 , error 32, ret -1005
2021-05-05T17:04:05UTC rmqClient Send returned -1 Err No 32, While Sending Len 66
2021-05-05T17:04:05UTC rmqClient Error Sending Data on Idx 19 Status -1005
2021-05-05T17:04:05UTC rmqClient Unable to Send Data for Id 19 , error 32, ret -1005
2021-05-05T17:04:10UTC rmqClient Send returned -1 Err No 32, While Sending Len 66
2021-05-05T17:04:10UTC rmqClient Error Sending Data on Idx 19 Status -1005
2021-05-05T17:04:10UTC rmqClient Unable to Send Data for Id 19 , error 32, ret -1005
2021-05-05T17:04:15UTC rmqClient Send returned -1 Err No 32, While Sending Len 66
2021-05-05T17:04:15UTC rmqClient Error Sending Data on Idx 19 Status -1005
2021-05-05T17:04:15UTC rmqClient Unable to Send Data for Id 19 , error 32, ret -1005
2021-05-05T17:04:20UTC rmqClient Send returned -1 Err No 32, While Sending Len 66
2021-05-05T17:04:20UTC rmqClient Error Sending Data on Idx 19 Status -1005
2021-05-05T17:04:20UTC rmqClient Unable to Send Data for Id 19 , error 32, ret -1005
2021-05-05T17:04:25UTC rmqClient Send returned -1 Err No 32, While Sending Len 66
2021-05-05T17:04:25UTC rmqClient Error Sending Data on Idx 19 Status -1005
2021-05-05T17:04:25UTC rmqClient Unable to Send Data for Id 19 , error 32, ret -1005
2021-05-05T17:04:30UTC rmqClient Send returned -1 Err No 32, While Sending Len 66
2021-05-05T17:04:30UTC rmqClient Error Sending Data on Idx 19 Status -1005
2021-05-05T17:04:30UTC rmqClient Unable to Send Data for Id 19 , error 32, ret -1005
2021-05-05T17:04:35UTC rmqClient Connect Returned  -1
2021-05-05T17:04:40UTC rmqClient Connect Returned  -1

$ esxcli network ip connection list | grep 15002
tcp        66       0  127.0.0.1:15002   127.0.0.1:25572     ESTABLISHED   2488670  newreno  vsfwd
tcp         0       0  127.0.0.1:25572   127.0.0.1:15002     ESTABLISHED   2465681  newreno  netcpa
tcp        79       0  127.0.0.1:15002   127.0.0.1:24691     CLOSE_WAIT    2488670  newreno  vsfwd
tcp         0       0  127.0.0.1:24691   127.0.0.1:15002     FIN_WAIT_2    2488678  newreno
tcp         0       0  127.0.0.1:15002   127.0.0.1:31347     CLOSE_WAIT    2488670  newreno  vsfwd
tcp         0       0  127.0.0.1:31347   127.0.0.1:15002     FIN_WAIT_2    2465681  newreno  netcpa
tcp         0       0  127.0.0.1:15002   0.0.0.0:0           LISTEN        2488670  newreno  vsfwd

Environment

VMware vSphere ESXi 7.0.x

Cause

This is caused by the vsfwd process coming into a deadlock situation.
The cause of the deadlock is a child process trying to save the password never returned from a system call.

Resolution

Currently there is no resolution for VMware ESXi 7.0u2a. 

Please upgrade to VMware ESXi 7.0u2c or later to prevent this issue.

Workaround:
The current workaround is to re-install ESXi.