vCenter service crashes randomly with error: "Panic: UUIDCreateRandomUnstylized: Random_Crypto failure"

Products

• VMware vCenter Server

Issue/Introduction

To avoid the vCenter service crash.

Symptoms:

• vCenter service crashes randomly with error " Panic:UUIDCreateRandomUnstylized: Random_Crypto failure"
• The vpxd.log will have similar entries to below:

 2019-11-25T22:39:38.386-08:00 warning vpxd[7F71B2047700] [Originator@6876 sub=Default] Failed to accept connection; <acceptor p:0x00007f70f443fcc0, h:182, <TCP '127.0.0.1:8085'>>, e: system:24(Too many open files)
2019-11-25T22:39:38.386-08:00 error vpxd[7F71B2047700] [Originator@6876 sub=SoapAdapter.HTTPService] accept failure N7Vmacore15SystemExceptionE(Too many open files)
--> [context]zKq7AVECAAAAALOS2wAOdnB4ZAAA4LArbGlidm1hY29yZS5zbwAAwhcbAB5aGACwcisAZWUjAJ1pJgBHeyYAY4gmALcwJAAj/CMACv8jAJcJLAHUcwBsaWJwdGhyZWFkLnNvLjAAAt2ODmxpYmMuc28uNgA=[/context] on stream <io_obj p:0x00007f71acb50bf0, h:-1, <TCP '0.0.0.0:0'>, <TCP '0.0.0.0:0'>>
2019-11-25T22:39:38.424-08:00 panic vpxd[7F71B0C1F700] [Originator@6876 sub=Default]
-->
--> Panic: UUIDCreateRandomUnstylized: Random_Crypto failure!
-->
--> Backtrace:
--> [backtrace begin] product: VMware VirtualCenter, version: 6.5.0, build: build-14389939, tag: vpxd, cpu: x86_64, os: linux, buildType: release
--> backtrace[00] libvmacore.so[0x002BB0F8]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
--> backtrace[01] libvmacore.so[0x001B18B6]: Vmacore::System::SystemFactoryImpl::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
--> backtrace[02] libvmacore.so[0x002B90A7]
--> backtrace[03] libvmacore.so[0x002B9182]: Vmacore::PanicExit(char const*)
--> backtrace[04] libvmacore.so[0x0018F915]

• In the  vpxd-profiler-xx.log  it shows high number of API calls "waitForUpdates" and total number of sessions from client software.

Note:The preceding log excerpts are only examples, date, time and environmental variables will vary depending on your environment.

Environment

VMware vCenter Server 6.7.x
VMware vCenter Server Appliance 6.5.x
VMware vCenter Server Appliance 6.7.x
VMware vCenter Server Appliance 6.0.x
VMware vCenter Server 6.0.x
VMware vCenter Server 6.5.x

Cause

Client software is using deprecated waitForUpdates calls instead of waitForUpdatesEx calls. Using the deprecated API by client  software  is causing timing out before the waitForUpdates calls are completing. Use of waitForUpdatesEx will allow introduction of a timeout smaller than the one used by the client or firewall.

Resolution

This is not a VMware bug. Please contact the 3rd party client software vendor for the fix.

Workaround:
To workaround:
1. Disable or remove the Client software plugin. For mor details on steps refer to VMware KB
2. Once the user account/offending IP is determined, take steps to avoid making API waitForUpdates call.

Additional Information

Impact/Risks:
Requires restart of vSphere Client.