VPXD repeatedly crashes without writing backtraces in the vpxd logs

Products

• VMware vCenter Server

Issue/Introduction

• VPXD repeatedly crashes without writing backtraces in the vpxd logs
• when reviewing the last vpxd logs before a crash, /var/log/vmware/vpxd/vpxd-<number>.log, repeated authentication errors can be seen like:

  <DATE/TIME> error vpxd[1173882] [Originator@6876 sub=UserDirectorySso opID=<operation_ID>] AcquireToken exception: N9SsoClient27InvalidCredentialsExceptionE(Authentication failed: Invalid credentials)
  --> [context]zKq7AVECAQAAABiabwEadnB4ZAAAowFTbGlidm1hY29yZS5zbwAA/gdCAB8vQwAYm0oBQHICbGlic3NvY2xpZW50LnNvAAG+eAIBTYICAX6DAgFRZgOCtGaEAnZweGQAgtbf1QGC3uvVAYIbvNQBgwgpNQFsaWJ2aW0tdHlwZXMuc28AgjEVggIETXUcbGlidm1vbWkuc28AgmjvXwKCsKCAAoL7sIACgu/DfwKC3pqAAgAE3DcAFzU4ALv/UAWwjgBsaWJwdGhyZWFkLnNvLjAABu/6D2xpYmMuc28uNgA=[/context]
  <DATE/TIME> error vpxd[1173882] [Originator@6876 sub=User opID=<operation_ID>] Failed to authenticate user <USER_ACCOUNT>

• This can also shown in sts log file in /var/log/vmware/sso/:

  <DATE/TIME> WARN sts[95:tomcat-http--49] [CorId=5e103580-070a-4f98-9a9c-96c9dc8bc233] [com.vmware.identity.idm.server.provider.vmwdirectory.VMwareDirectoryProvider] Failed to authenticate using SRP binding
  com.vmware.identity.interop.ldap.InvalidCredentialsLdapException: Invalid credentials

Environment

VMware vCenter Server 8.0.3 

Cause

There is a dangling Session pointer in the the code responsible to cache the user data. When a new token is preferred, the code tries to update the expiration time of all the sessions, leading to dereferencing the dangling pointers, which accumulate in the vpxd memory and lead to the service crashing.

Resolution

This issue has been fixed in vCenter Server 8.0 Update 3e (8.0.3 Patch 5), if you are experiencing similar symptom please ensure to update vCenter Server to this version.

As a workaround, identify the user and the source of the failing logins and stop whichever external application is attempting them from doing so, by either disabling it, or by updating it to use valid credentials.