NSX Controller disconnected or isolates intermittently
search cancel

NSX Controller disconnected or isolates intermittently

book

Article ID: 321339

calendar_today

Updated On:

Products

VMware NSX Networking

Issue/Introduction

This article provides work around solution to NSX Controllers frequent isolation or disconnection issues.

Symptoms:
  • Running the show log command on the NSX Manager console reports entries similar to:

    2015-06-05 03:42:08.236 GMT WARN NVPInactiveNodeCheck RestTemplate:478 - GET request for "https://10.30.14.140:443/ws.v1/control-cluster" resulted in 503 (Service Unavailable); invoking error handler
    2015-06-05 03:42:08.236 GMT ERROR NVPInactiveNodeCheck NvpRestClientManagerImpl:700 - nvp controller node fails: org.springframework.web.client.HttpServerErrorException: 503 Service Unavailable


    Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.


Environment

VMware NSX for vSphere 6.2.x
VMware NSX for vSphere 6.1.x
VMware NSX for vSphere 6.0.x

Cause

This issue occurs when the NSX Controller is isolated due to a known bug that appears during IPSEC rekeying. The same bug may appear when you restart IPSEC. (This usually happens when you reboot the appliance or restart the controller)

Resolution

This issue is resolved in VMware NSX for vSphere 6.1.5 and 6.2.1, available at VMware Downloads.

To work around the issue, make an API call to disable IPsec VPN for controller to controller communications.

Method: PUT
URL: https://NSX-Manager-IP/api/2.0/vdn/controller/node
Body:
<controllerNodeConfig>
<ipSecEnabled>false</ipSecEnabled>
</controllerNodeConfig>


For more information on how to make API calls to the NSX Manager, see the Using the NSX REST API section in the VMware NSX for vSphere API Guide.


Additional Information

If an NSX controller cluster is set to allow controller-to-controller communications in the clear (IPsec is disabled), and then later the administrator re-enables IPsec communication, one or more controllers may become isolated from the cluster majority due to a mismatched pre-shared key ("PSK"). When this occurs, the NSX API may become unable to change the IPsec settings of the controllers.

If you encounter this, follow these steps to address this issue:
  1. Disable IPsec using the NSX API:

    Method: PUT
    URL: https://NSX-Manager-IP/api/2.0/vdn/controller/node
    Body:
    <controllerNodeConfig>
    <ipSecEnabled>false</ipSecEnabled>
    </controllerNodeConfig>

     
  2. Re-enable IPsec using the NSX API:

    Method: PUT
    URL: https://NSX-Manager-IP/api/2.0/vdn/controller/node
    Body:
    <controllerNodeConfig>
    <ipSecEnabled>true</ipSecEnabled>
    </controllerNodeConfig>
To view the current IPsec status, use the show control-cluster network ipsec status command in VMware NSX for vSphere 6.1.5 and 6.2.1 or later.

Use these best practices to avoid this issue:
  • Always use the NSX API to disable IPsec. Using the NSX Controller CLI to disable IPsec is unsupported.
  • Always verify that all controllers are active before you use the API to change the IPsec setting.
To be alerted when this document is updated, click the Subscribe to Article link in the Actions box.

Collecting diagnostic information for VMware NSX for vSphere 6.x
NSX Controller 间歇性断开连接或隔离
VMware NSX for vSphere 6.x Controller が断続的に切断される