Upgrade/patch to vCenter 7.0 U3i or later fails due to STS service not starting
search cancel

Upgrade/patch to vCenter 7.0 U3i or later fails due to STS service not starting

book

Article ID: 324288

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

Attempting to upgrade/patch to vCenter 7.0 U3i or later fails. The /var/log/vmware/applmgmt/PatchRunner.log shows entries similar to the following: 
2022-12-27T19:47:18.759Z ERROR vmware_b2b.patching.phases.patcher Patch hook Patch got unhandled exception.
Traceback (most recent call last):
  File "/storage/updatemgr/software-updaterqx0aqjd/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 203, in patch
    _patchComponents(ctx, userData, statusAggregator.reportingQueue)
  File "/storage/updatemgr/software-updaterqx0aqjd/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 84, in _patchComponents
    _startDependentServices(c)
  File "/storage/updatemgr/software-updaterqx0aqjd/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 53, in _startDependentServices
    serviceManager.start(depService)
  File "/storage/updatemgr/software-updaterqx0aqjd/stage/scripts/patches/libs/sdk/service_manager.py", line 901, in wrapper
    return getattr(controller, attr)(*args, **kwargs)
  File "/storage/updatemgr/software-updaterqx0aqjd/stage/scripts/patches/libs/sdk/service_manager.py", line 794, in start
    super(VMwareServiceController, self).start(serviceName)
  File "/storage/updatemgr/software-updaterqx0aqjd/stage/scripts/patches/libs/sdk/service_manager.py", line 665, in start
    raise IllegalServiceOperation(errorText)
service_manager.IllegalServiceOperation: Service cannot be started. Error: Error executing start on service sts. Details {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "sts"
            ],
            "localized": "An error occurred while starting service 'sts'"
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
}
Service-control failed. Error: {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "sts"
            ],
            "localized": "An error occurred while starting service 'sts'"
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
}


2022-12-27T19:47:19.762Z WARNING root stopping status aggregation...
2022-12-27T19:47:19.764Z ERROR __main__ Patch vCSA failed
The /var/log/vmware/sso/sts-runtime.log.stderr will show entries similar to the following: 
Starting service process with pid: 62572.
Picked up JAVA_TOOL_OPTIONS: -Xms32M -Xmx128M     -Dcom.sun.org.apache.xml.internal.security.ignoreLineBreaks=true     -Dorg.apache.xml.security.ignoreLineBreaks=true
java.lang.Error: org.apache.catalina.LifecycleException: Failed to initialize connector [Connector[com.vmware.identity.tomcat.ClientAuthHttp11NioProtocol-3128]]
        at org.apache.catalina.startup.Catalina.load(Catalina.java:649)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:305)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:475)
Caused by: org.apache.catalina.LifecycleException: Failed to initialize connector [Connector[com.vmware.identity.tomcat.ClientAuthHttp11NioProtocol-3128]]
        at org.apache.catalina.core.StandardService.initInternal(StandardService.java:578)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
        at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:874)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
        at org.apache.catalina.startup.Catalina.load(Catalina.java:646)
        ... 6 more
The catalina log under /var/log/vmware/sso/tomcat will show entries similar to the following: 
2022-12-27T19:47:17.420Z SEVE org.apache.catalina.core.StandardService Failed to initialize connector [Connector[com.vmware.identity.tomcat.ClientAuthHttp11NioProtocol-3128]]
org.apache.catalina.LifecycleException: Protocol handler initialization failed
        at org.apache.catalina.connector.Connector.initInternal(Connector.java:1114)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
        at org.apache.catalina.core.StandardService.initInternal(StandardService.java:571)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
        at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:874)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
        at org.apache.catalina.startup.Catalina.load(Catalina.java:646)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:305)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:475)
Caused by: java.lang.IllegalArgumentException: the trustAnchors parameter must be non-empty
        at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:108)
        at org.apache.tomcat.util.net.AbstractJsseEndpoint.initialiseSsl(AbstractJsseEndpoint.java:72)
        at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:205)
        at org.apache.tomcat.util.net.AbstractEndpoint.bindWithCleanup(AbstractEndpoint.java:1221)
        at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1234)
        at org.apache.tomcat.util.net.AbstractJsseEndpoint.init(AbstractJsseEndpoint.java:230)
        at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:633)
        at org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:80)
        at org.apache.catalina.connector.Connector.initInternal(Connector.java:1111)
        ... 12 more
Caused by: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
        at java.security.cert.PKIXParameters.setTrustAnchors(PKIXParameters.java:200)
        at java.security.cert.PKIXParameters.<init>(PKIXParameters.java:157)
        at java.security.cert.PKIXBuilderParameters.<init>(PKIXBuilderParameters.java:130)
        at org.apache.tomcat.util.net.SSLUtilBase.getParameters(SSLUtilBase.java:496)
        at org.apache.tomcat.util.net.SSLUtilBase.getTrustManagers(SSLUtilBase.java:427)
        at org.apache.tomcat.util.net.SSLUtilBase.createSSLContext(SSLUtilBase.java:246)
        at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:106)
        ... 20 more


Environment

VMware vCenter Server 7.0.3

Cause

This can be caused by the file /usr/lib/vmware-sso/vmware-sts/conf/clienttrustCA.pem being empty.

Resolution

Create a trusted client CA store and populate the clienttustCA pem file with the CA certificates that signed the smart card certificates.

Once the clienttrustCA.pem file is updated, select the resume option on the "Installation Failed" dialog to continue the patch installation.

Reference:

This store contains the trusted issuing CA's certificates for client certificate. The client here is the browser from which the smart card process prompts the end user for information.

Create a certificate store on the vCenter Server using the exact path and PEM name, /usr/lib/vmware-sso/vmware-sts/conf/clienttrustCA.pem.

For a single certificate:

cd /usr/lib/vmware-sso/ openssl x509 -inform PEM -in xyzCompanySmartCardSigningCA.cer > /usr/lib/vmware-sso/vmware-sts/conf/clienttrustCA.pem

For multiple certificates:

cd /usr/lib/vmware-sso/ openssl x509 -inform PEM -in xyzCompanySmartCardSigningCA.cer >> /usr/lib/vmware-sso/vmware-sts/conf/clienttrustCA.pem

See https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/7-0/vsphere-authentication-7-0.html

Powered by Wolken Software