vCenter services fails to start with error "java.security.KeyStoreException: VKS not found"
Article ID: 423675
Updated On:
Products
Issue/Introduction
Symptoms:
- vCenter lookupsvc and other associated services fails to start with below error:
# root@vcenter: service-control --start --all
Service-control failed. Error: Failed to start services in profile ALL. RC=1, stderr=Failed to start trustmanagement, sps, vapi-endpoint, vpxd, lookupsvc, sca, cis-license, eam, vpxd-svcs, sts services. Error: Operation timed out
- From vCenter log
/var/log/vmware/sso/tomcat/catalina.log, below errors are seen:
YYYY-MM-DDTHH:MM:SS.Z WARN org.apache.catalina.security.SecurityListener No umask setting was found in system property [org.apache.catalina.security.SecurityListener.UMASK]. However, it appears Tomcat is running on a platform that supports umask. The system property is typically set in CATALINA_HOME/bin/catalina.sh. The Lifecycle listener org.apache.catalina.security.SecurityListener (usually configured in CATALINA_BASE/conf/server.xml) expects a umask at least as restrictive as [0xx7]YYYY-MM-DDTHH:MM:SS.Z INFO org.apache.coyote.http11.Http11NioProtocol Initializing ProtocolHandler ["http-nio-127.x.x.x-7xx0"]YYYY-MM-DDTHH:MM:SS.Z INFO org.apache.coyote.http11.Http11NioProtocol Initializing ProtocolHandler ["https-Vecs Aware JSSE-nio-127.x.x.x-7xx4"]YYYY-MM-DDTHH:MM:SS.Z SEVE com.vmware.identity.tomcat.VECSAwareSSLImplementation Failed to load VECS keystore 'MACHINE_SSL_CERT'java.security.KeyStoreException: VKS not found at java.security.KeyStore.getInstance(KeyStore.java:851) at com.vmware.identity.tomcat.VECSAwareSSLImplementation.getVecsKeystore(VECSAwareSSLImplementation.java:110) at com.vmware.identity.tomcat.VECSAwareSSLImplementation.getSSLUtil(VECSAwareSSLImplementation.java:46) at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:96) at org.apache.tomcat.util.net.AbstractJsseEndpoint.initialiseSsl(AbstractJsseEndpoint.java:71) at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:228) at org.apache.tomcat.util.net.AbstractEndpoint.bindWithCleanup(AbstractEndpoint.java:1373) at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1386) at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:663) at org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:80) at org.apache.catalina.connector.Connector.initInternal(Connector.java:1046) at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:122) at org.apache.catalina.core.StandardService.initInternal(StandardService.java:522) at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:122) at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:986) at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:122) at org.apache.catalina.startup.Catalina.load(Catalina.java:690) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:302) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:475)Caused by: java.security.NoSuchAlgorithmException: VKS KeyStore not available
- Validate all the certificates are valid in vCenter using below command:
# for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | grep -ie "Alias" -ie "Not After";done;
- Verify that no disk partition full issue with the command
"df -h"
Environment
VMware vCenter 8.x
Resolution
Reboot the vCenter to fix the issue.
Feedback
