NSX Manager unavailability due to FIPS validation failure

Products

• VMware NSX
• VMware Cloud Foundation

Issue/Introduction

• The NSX Manager UI is not responding.  No error is provided, but the interface does not load.
• SSH to the manager is successful and you can login to the CLI.
• As the admin user on the CLI, the services appear to be running when executing:
  

  get services

• As the admin user on the CLI, the status to the other managers is "Connected" when executing:

  get cluster status

• As the root user on the CLI, execute:

  grep "proportionate test failed" syslog

  This situation applies if there are matching results that include org.bouncycastle.crypto.fips.FipsOperationError: proportionate test failed

Environment

VMware NSX

VMware Cloud Foundation

Cause

The BouncyCastle FIPS-certified cryptographic module failed its continuous self-testing requirements (proportionate test failed). This is a safety mechanism that puts the module into an error state, preventing SSL/TLS handshakes and service initialization. This often occurs under heavy load due to lack of sufficient entropy from the underlying OS.

Resolution

Fixed in NSX release 4.2.3.3 and higher. See Download Broadcom products and software for steps to download this release.

Workaround: If an immediate upgrade is not possible, perform a rolling reboot of the NSX Manager nodes to restore cryptographic services and reset the FIPS self-test state.

Verification:

1. Log in to the NSX Manager CLI as root.
2. Execute the following command to confirm the error: grep "proportionate test failed" /var/log/syslog 
3. If results contain org.bouncycastle.crypto.fips.FipsOperationError: proportionate test failed, the manager node requires a reboot.