In NSX Cross vCenter environment the BGP password is not replicated to newly promoted Primary manager

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

• During failover or failback in an NSX Cross vCenter environment, the BGP relationship may not come up.
• In the NSX Edge logs (show log follow):

2019-05-29T16:55:39+00:00 NSX-edge-1-0 kernel[]: [default]:  [kern.info] TCP: MD5 Hash failed for (192.168.119.1, 25307)->(192.168.119.254, 179)
2019-05-29T16:55:40+00:00 NSX-edge-1-0 kernel[]: [default]:  [kern.info] TCP: MD5 Hash failed for (192.168.119.1, 25307)->(192.168.119.254, 179)
2019-05-29T16:55:42+00:00 NSX-edge-1-0 kernel[]: [default]:  [kern.info] TCP: MD5 Hash failed for (192.168.119.1, 25307)->(192.168.119.254, 179)
2019-05-29T16:55:46+00:00 NSX-edge-1-0 kernel[]: [default]:  [kern.info] TCP: MD5 Hash failed for (192.168.119.1, 25307)->(192.168.119.254, 179)
2019-05-29T16:55:54+00:00 NSX-edge-1-0 kernel[]: [default]:  [kern.info] TCP: MD5 Hash failed for (192.168.119.1, 25307)->(192.168.119.254, 179)

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX for vSphere 6.3.x
VMware NSX for vSphere 6.4.x

Cause

There is an issue with NSX Cross vCenter 6.3.x and 6.4.x where the NSX manager fails to copy the password over from the primary NSX manager to the secondary NSX manager, which will now be the new primary NSX manager.
The means the BGP relationship cannot form and is stuck in connect/active state.
To validate if the BGP password was not sync'ed to the secondary NSX manager (now new primary NSX manager), the following API can be used to retrieve the BGP password. Be aware this returns the password in clear text.
GET /api/4.0/edges/<edge-id>?global_published_config

Resolution

This is resolved in NSX 6.4.4.

Workaround:
If you are unable to upgrade, you can use the following workaround:

• Manually re-entering the same BGP password and save, will allow the BGP relationships to form again.
• Note: It is advised to include this step in any failover/failback procedures, just after the new UDLR control VM is deployed.

If BGP password is unknown, the following API can be used to retrieve the BGP password. Be aware this returns the password in cleartext:

• Note: This API can only be run on the primary manager, so if the failover has already occurred and the new primary does not have the password, you will not see the BGP password.
• GET /api/4.0/edges/<edge-id>?global_published_config

Additional Information

Impact/Risks:
This issue can lead to a dataplane impact, as routes are not learned and published.