The VPXD service on the vCenter server crashes when an account logs in using an existing authenticated session

Products

• VMware vCenter Server 8.0

Issue/Introduction

• VPXD service on the vCenter server crashes intermittently and generates core.vpxd.worker.##### files in /storage/core/ partition.
  Storage utilization of the /storage/core/ partition increases and may generate alarms in vCenter after the utilization exceeds 70%.
  In /var/log/vmware/vmon/vmon.log, you may find entries similar to:

YYYY-MM-DDTHH:MM:SS.821Z Wa(03) host-#### <vpxd> Service exited. Exit code 1
YYYY-MM-DDTHH:MM:SS.821Z Wa(03) host-####<vpxd> Service exited unexpectedly. Crash count 0. Taking configured recovery action.
YYYY-MM-DDTHH:MM:SS.821Z In(05) host-####<vpxd> Restarting service.

• In the /var/log/vMonCoredumper.log, you may find entries similar to:

YYYY-MM-DDTHH:MM:SS.317Z In(05) host-#### Notify vMon about vpxd-worker dumping core. Pid : ####
YYYY-MM-DDTHH:MM:SS.329Z In(05) host-#### Successfully notified vMon.
YYYY-MM-DDTHH:MM:SS.792Z In(05) host-#### Successfully generated core file /var/core/core.vpxd-worker.####.

• At the time of the service crash, in /var/log/vmware/vpxd/vpxd-###.log, entries related to login attempts are observed:

YYYY-MM-DDTHH:MM:SS info vpxd[2858939] [Originator@6876 sub=vpxLro opID=###### Authz-e2] [VpxLRO] -- BEGIN lro-909100 -- AuthorizationManager -- vim.AuthorizationManager.hasUserPrivilegeOnEntities -- ########-####-####-####-############(########-####-####-####-############)
YYYY-MM-DDTHH:MM:SS info vpxd[2858939] [Originator@6876 sub=UserDirectorySso opID=###### Authz-e2] GetUserInfoInternal(Domain\Username, false) res: Domain\Username
YYYY-MM-DDTHH:MM:SS info vpxd[2858939] [Originator@6876 sub=vpxLro opID=###### Authz-e2] [VpxLRO] -- FINISH lro-909100
YYYY-MM-DDTHH:MM:SS info vpxd[2858710] [Originator@6876 sub=UserDirectorySso opID=Run-Http2ServerSession-41] GetUserInfoInternal(Domain\Username, false) res: Domain\Username
YYYY-MM-DDTHH:MM:SS info vpxd[2858710] [Originator@6876 sub=AuthorizeManager opID=Run-Http2ServerSession-41] [Auth]: User Domain\Username

• Before the above login attempt, the Journal logs show multiple failed login attempts for the same user:

> journalctl -b 0 | grep AlreadyAuthenticatedSessionEvent

Event [43805188] [1-1] [YYYY-MM-DDTHH:MM:SS.416929Z] [vim.event.AlreadyAuthenticatedSessionEvent] [info] [Domain\Username] [] [43805188] [User cannot logon since the user is already logged on]
Event [43805189] [1-1] [YYYY-MM-DDTHH:MM:SS.450867Z] [vim.event.AlreadyAuthenticatedSessionEvent] [info] [Domain\Username] [] [43805189] [User cannot logon since the user is already logged on]
Event [43805190] [1-1] [YYYY-MM-DDTHH:MM:SS.486669Z] [vim.event.AlreadyAuthenticatedSessionEvent] [info] [Domain\Username] [] [43805190] [User cannot logon since the user is already logged on] 

Environment

VMware vCenter Server 8.x

Cause

This issue occurs when there is a login attempt from a client on an already authenticated session.
In this scenario, the VPXD service crashes due to a dangling session pointer in session cache management.

• On a failed login attempt, an incomplete cleanup ends up leaving a unreferenced pointer in the session data structure.
• On subsequent logins, an attempt to update the expiration time for all sessions accesses this dangling pointer which leads to a crash.

Resolution

This issue has been fixed in vCenter Server 8.0 Update 3e Build 24674346.

---

Workaround:

To prevent this issue from occurring, identify the IP address of the solution which is attempting logins to the vCenter server for an already authentication session.
Once the username is identified from /var/log/vmware/vpxd.log, review journal logs to determine the IP address of the client. It is recommended to temporarily disable logins from this specific client to prevent further crashes.

Sample journal logs:
Event [43805072] [1-1] [YYYY-MM-DDTHH:MM:SS.98434Z] [vim.event.UserLoginSessionEvent] [info] [Domain\Username] [] [43805072] [User Domain\Username@#.#.#.# logged in as JAX-WS RI 2.3.1 svn-revision#]

To temporarily mitigate VPXD from crashing, the below workaround can be applied with caution:

1) Stop vpxd service.
service-control --stop vpxd

2) Edit the /etc/vmware-vpx/vpxd.cfg file, locate the <vpxd> section and ensure the following setting is present. If it is not present, add it manually:
<authorize><sessionCanOutliveToken>true</sessionCanOutliveToken></authorize>

3) Start vpxd service.
service-control --start vpxd

This workaround prevents the vCenter from crashing when handling authentication errors.

Note: The above workaround will keep the sessions alive even after the token has expired. This could lead to a security issues and hence use it with caution and only as a temporary workaround.