"Failed to initialize Secure Token Server""Installation of component VCSServiceManager failed with error code '1603'" while upgrading VC to 6.7U3

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• Windows vCenter Server upgrade fails with "Installation of component VCSServiceManager failed with error code '1603'"

Traceback (most recent call last):
 File "C:\Program Files\VMware\vCenter Server\firstboot\vmidentity-firstboot.py", line 1639, in main
  vmidentityFB.boot()
 File "C:\Program Files\VMware\vCenter Server\firstboot\vmidentity-firstboot.py", line 345, in boot
  self.checkSTS(self.__stsRetryCount, self.__stsRetryInterval)
 File "C:\Program Files\VMware\vCenter Server\firstboot\vmidentity-firstboot.py", line 1179, in checkSTS
  raise Exception('Failed to initialize Secure Token Server.')
Exception: Failed to initialize Secure Token Server.

• You will see the similar entries in the logs files as mentioned below C:ProgramData/VMware/vCenterServer/logs/sso/vmware-sts-idmd.log:

Caused by: java.lang.ClassNotFoundException: org.apache.xml.serializer.TreeWalker
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:1.8.0_212]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_212]
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) ~[?:1.8.0_212]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_212]
        at org.apache.xalan.processor.TransformerFactoryImpl.newTransformer(TransformerFactoryImpl.java:823) ~[xalan-2.7.1.jar:?]
        at com.sun.xml.internal.messaging.saaj.util.transform.EfficientStreamingTransformer.materialize(EfficientStreamingTransformer.java:106) ~[?:1.8.0_212]
        at com.sun.xml.internal.messaging.saaj.util.transform.EfficientStreamingTransformer.setOutputProperty(EfficientStreamingTransformer.java:190) ~[?:1.8.0_212]
        at com.sun.xml.internal.messaging.saaj.soap.impl.EnvelopeImpl.output(EnvelopeImpl.java:260) ~[?:1.8.0_212]
        at com.sun.xml.internal.messaging.saaj.soap.impl.EnvelopeImpl.output(EnvelopeImpl.java:302) ~[?:1.8.0_212]
        at com.sun.xml.internal.messaging.saaj.soap.SOAPPartImpl.getContentAsStream(SOAPPartImpl.java:311) ~[?:1.8.0_212]
        at com.sun.xml.internal.messaging.saaj.soap.MessageImpl.getHeaderBytes(MessageImpl.java:1015) ~[?:1.8.0_212]
        at com.sun.xml.internal.messaging.saaj.soap.MessageImpl.saveChanges(MessageImpl.java:1166) ~[?:1.8.0_212]
        at com.sun.xml.internal.ws.api.message.saaj.SAAJFactory.readAsSOAPMessage(SAAJFactory.java:277) ~[?:1.8.0_212]
        at com.sun.xml.internal.ws.api.message.saaj.SAAJFactory.readAsSAAJ(SAAJFactory.java:197) ~[?:1.8.0_212]
        at com.sun.xml.internal.ws.api.message.saaj.SAAJFactory.read(SAAJFactory.java:186) ~[?:1.8.0_212]
        at com.sun.xml.internal.ws.message.AbstractMessageImpl.toSAAJ(AbstractMessageImpl.java:217) ~[?:1.8.0_212]
        at com.sun.xml.internal.ws.api.message.MessageWrapper.readAsSOAPMessage(MessageWrapper.java:156) ~[?:1.8.0_212]
        at com.sun.xml.internal.ws.client.dispatch.SOAPMessageDispatch.toReturnValue(SOAPMessageDispatch.java:89) ~[?:1.8.0_212]

Environment

VMware vCenter Server 6.7.x

Cause

•  During minor update or major upgrade some jar files in install directory may not be cleaned up properly, and left behind. This results in these jars being loaded into Java VM process when they should not be, which is causing internal java runtime exceptions, as various components started to conflict.

• In this instance the upgrade was failing because of a stale jar file xalan-2.7.1.jar from previous updates.

Resolution

To solve this issue:

1. Navigate to the path InstallDirectory:\Program Files\VMware\vCenter Server\VMware Identity Services
2. Detect the jar files found from the logs
3. Stop the service VMware Identity Services, VMware Secure Token Service, and VMware Lifecycle Service
4. Move the identified jar file out of the directory
5. Start the services
6. Perform the upgrade