NCP fails to create a new logical segment when there are problems communicating with one of the NSX Managers

Products

• VMware Tanzu Application Service

Issue/Introduction

App Deployment may fail with the following error

external networker up: exit status 1

 

NCP logs will show the following error code 8212.  Two important symptoms in this error show there is a problem with NCP communicating with NSX Manager.  The http status is "BAD_REQUEST" and the cidr is "null".  In this case one of 3 NSX managers would not respond successfully to HTTP requests. 

'httpStatus': 'BAD_REQUEST', 'error_code': 8212, 'module_name': 'NsxSwitching service', 'error_message': 'Failed to get a valid IP from IpPool IpPool/GUID with cidr null.'}.

Cause

After a temporary network or storage outage NSX Manager was not able to recover as it encountered a known issue preventing the systemDownHandler from restarting the CCP service, which is fixed in NSX 4.2.0 and later versions.  See also a TKGi symptom here

Resolution

The resolution in this case was to reboot the NSX managers 1 by 1. One of 3 NSX managers would not respond successfully to HTTP requests resulting in logical port segment creation failures.