Secure Token Service (vmware-stsd) in vCenter Server 6.7 U1 crashes with core.jsvc.xxx files causing /storage/core to fill up

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• Logging into vCenter Server 6.7 Update 1 fails.
• The virtual hardware version of the vCenter Appliance has been manually upgraded to Version 13 or higher since it was installed.
• vmware-stsd service crashes in environments configured with Integrated Windows Authentication (IWA) as an Identity Source.
• Files named core.jsvc.xxx are generated until /storage/core directory is full. 
• In vmware-sts-idmd.log file, you see entries similar to:

[2018-11-02T13:28:42.168-07:00 IDM Shutdown INFO ] [IdmServer] Stopping IDM Server...
[2018-11-02T13:28:42.523-07:00 IDM Shutdown INFO ] [IdmServer] IDM Server has stopped
[2018-11-02T13:29:38.270-07:00 IDM Startup INFO ] [IdmServer] Starting IDM Server...
[2018-11-02T13:29:38.272-07:00 IDM Startup INFO ] [IdmServer] IDM Server has started
[2018-11-02T13:39:40.913-07:00 IDM Shutdown INFO ] [IdmServer] Stopping IDM Server...
[2018-11-02T13:39:40.913-07:00 IDM Shutdown INFO ] [IdmServer] IDM Server has stopped

 

• • In /var/log/vmware/sso/utils/vmware-stsd.err, you see entries similar to:

Nov 02, 2018 1:29:40 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 663 ms
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/vmware-sso/vmware-sts/webapps/ROOT/WEB-INF/lib/log4j-slf4j-impl-2.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/vmware-sso/vmware-sts/webapps/ROOT/WEB-INF/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Nov 02, 2018 1:29:50 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 10097 ms
Service killed by signal 11

If in PSC /storage/core are filled continuously, see PSC /storage/core gets filled in vCenter Server 6.5.

Environment

VMware vCenter Server Appliance 6.7.x

Cause

This issue occurs when the virtual hardware version of vCenter Appliance has been updated to 13 or higher.

Note: VMware does not support updating the hardware version of the vCenter Server Appliance.

Resolution

To resolve this issue:

The virtual hardware of the vCenter Server or PSC must be reverted to its original version. To do this, backup the appliance and restore it using the built-in backup and restore functionality in vCenter.

1. Delete the core dump files under  /storage/core
2. Use File-Based Backup and Restore to take the backup, see File-Based Backup and Restore of vCenter Server Appliance
3. Power off the Virtual Machine.
4. Use the installer (must be the same build) to restore the appliance to a new virtual machine.

Workaround:
In many cases, it has been discovered that changing the identity source from IWA to AD over LDAP will temporarily work around the problem. 

To work around this issue:

1. Remove the IWA Identity Source configuration and re-add it as AD over LDAP or AD over LDAPS
2. Remove the PSC from the Active Directory domain.
3. Reboot the PSC

Please note that the vCenter Server Appliance must still be reverted to its original hardware version to be in a supported configuration. 

Additional Information

Impact/Risks:
stsd crashes generate core dumps, which fills up the /storage/core directory and eventually blocks authentication to vCenter Server.