SPS Service crashes with core.jsvc.xxx files causing /storage/core to fill up in vCSA 6.7 U2

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• vmware-sps, vmware-updatemgr and vmware-content-library services crashes in environment.
• Files named core.jsvc.xxx are generated until /storage/core directory is full.
• The virtual hardware version of the vCenter Appliance has been manually upgraded.
• In sps.log file, you see entries similar to:

yyyy-mm-ddThh:mm:ss.900-05:00 [main] ERROR opId=sps-Main-632228-208 com.vmware.vim.storage.common.task.retry.CallableRetryDecorator - Caught exception -
com.vmware.vim.storage.common.serviceclient.ls.LsException: Error while looking up service info
        at com.vmware.vim.storage.common.serviceclient.ls.LsException.fromEx(LsException.java:58)
        at com.vmware.vim.storage.common.serviceclient.ls.impl.ExternalServiceRegistryImpl.findOne(ExternalServiceRegistryImpl.java:186)
        at com.vmware.vim.storage.common.serviceclient.ls.impl.ExternalServiceRegistryImpl.findVc(ExternalServiceRegistryImpl.java:203)
        at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientLifeCycle.findServiceUri(VpxdClientLifeCycle.java:77)
        at com.vmware.vim.storage.common.serviceclient.ConnectionInitializationTask$CallableTemplate.call(ConnectionInitializationTask.java:116)
        at com.vmware.vim.storage.common.task.retry.CallableRetryDecorator.call(CallableRetryDecorator.java:64)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientManagerImpl.initialize(VpxdClientManagerImpl.java:106)
        at com.vmware.sps.StorageMain.commonInitialization(StorageMain.java:188)
        at com.vmware.sps.StorageMain.main(StorageMain.java:67)
Caused by: com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 503
        at com.vmware.vim.vmomi.client.common.Response$Status.getStatus(Response.java:58)
        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:156)
        at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:53)
        at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:226)
        at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:106)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:629)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:610)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:360)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:311)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:184)
        at com.sun.proxy.$Proxy35.list(Unknown Source)

Environment

VMware vCenter Server Appliance 6.7.x

Cause

This issue occurs when the virtual hardware version of vCenter Appliance has been updated to 13 or higher

Resolution

Modification of vCSA virtual hardware is unsupported.

Workaround:
To recover from this problem:

The virtual hardware of the vCenter Server or PSC must be reverted to its original version.

1. Delete the core dump files under  /storage/core
2. Use File-Based Backup and Restore to take the backup, see File-Based Backup and Restore of vCenter Server Appliance
3. Power off the Virtual Machine.
4. Use the installer (must be the same build) to restore the appliance to a new virtual machine.