SDDC manager UI failed to load due to corrupted cacerts file

Products

VMware Cloud Foundation

Issue/Introduction

Symptoms:

SDDC Manager UI is not loading
The 'commonsvcs' service fails to start
The following error is observed in the /var/log/vmware/vcf/sddc-manager-ui-app/sddcManagerServer.log file:

YYYY-MM-DDTHH:MM:SS.844+0000 WARN [4394c04c2fd24b43, 5f03512217194914] [services/pscUtils.js, init-pscs, attemptPSCInitWithRetry:106]
100.109: VError: PSC Initilization attempt "12" failed: Failed to initiate PSC: Unable to initialize psc inventory data: Failed to fetch results from /inventory/pscs api: 502 - "<html>\n<head><title>502 Bad Gateway</title></head>\n<body>\n<center><h1>502 Bad Gateway</h1></center>\n<hr><center>nginx</center>\n</body>\n</html>\n"
at Object.initializationPscError (/opt/vmware/vcf/sddc-manager-ui-app/server/src/errors/VCFError.js:100:5)
at attemptPSCInitWithRetry (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/pscUtils.js:99:46)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Error Info: {"retryCount":12,"status":403,"errorModule":100,"errorCode":109}

The following error may be observed in the /var/log/vmware/vcf/commonsvcs/vcf-commonsvcs.log file:

YYYY-MM-DDTHH:MM:SS.087+0000 ERROR [common,0000000000000000,0000] [c.v.e.s.c.util.LocalProcessService,main] Local Command Failed with exit value 1.
Output Logs :
Error Logs are stored at LocalProcess ERROR: 2025-05-05 15:05:11 - /bin/cp: error writing '/usr/lib/jvm/openjdk-java17-headless.x86_64/lib/security/cacerts': No space left on device
YYYY-MM-DDTHH:MM:SS.622+0000 ERROR [common,0000000000000000,0000] [o.s.boot.SpringApplication,main] Application run failed
org.springframework.context.ApplicationContextException: Unable to start web server; nested exception is org.springframework.boot.web.server.WebServerException: Unable to start embedded Tomcat
instantiate [javax.net.ssl.TrustManager]: Factory method 'trustManager' threw exception; nested exception is com.vmware.vcf.secure.errors.VcfSecurityException: Failed to initialize trust manager
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:800)

The following errors may also be observed in the /var/log/vmware/vcf/commonsvcs/vcf-commonsvcs.log file:

2025-11-14T00:36:52.935+0000 ERROR [common,0000000000000000,0000] [o.s.b.w.e.tomcat.TomcatStarter,main] Error starting Tomcat context. Exception: org.springframework.
beans.factory.UnsatisfiedDependencyException. Message: Error creating bean with name 'com.vmware.vcf.functionality.toggle.interceptor.FunctionalityToggleApiFilter' de
fined in URL [jar:file:/opt/vmware/vcf/commonsvcs/lib/vcf-commonsvcs.jar!/BOOT-INF/lib/vcf-common-core/vcf-functionality-toggle/libvcf-functionality-toggle.jar!/com/v
mware/vcf/functionality/toggle/interceptor/FunctionalityToggleApiFilter.class]: Unsatisfied dependency expressed through constructor parameter 1:

...

Failed to instantiate [com.vmware.cloud.foundation.rest.commonsvcs.service.SecuritySettingsApi]: Factory method 'getSecuritySettingsApi' threw exception with message: null
YYYY-MM-DDTHH:MM:SS.113+0000 INFO [common,0000000000000000,0000] [o.a.catalina.core.StandardService,main] Stopping service [Tomcat]
YYYY-MM-DDTHH:MM:SS+0000 WARN [common,0000000000000000,0000] [o.s.b.w.s.c.AnnotationConfigServletWebServerApplicationContext,main] Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Unable to start web server

...

Caused by: java.security.KeyStoreException: problem accessing trust store

Environment

VMware Cloud Foundation 5.x

Cause

When the / partition (e.g. root partition) fills up, changes to the cacerts file fail, resulting in corruption:

lrwxrwxrwx 1 root root 64 Mon DD HH:MM cacerts.jks -> /usr/lib/jvm/openjdk-java17-headless.x86_64/lib/security/cacerts
-rw-r--r-- 1 root root 192K Mon DD HH:MM cacerts

Resolution

Take a snapshot of the SDDC Manager VM
SSH to SDDC Manager VM using vcf account and su to root
Rebuild the cacerts file or copy one from a running SDDC manager:
Check if the cacerts file exists in the following directory: /storage/alt_root/usr/lob/jvm/openjdk-java17-headless.x86_64/lib/security/
find / -iname cacerts
Copy the file to the /etc/alternatives/jre/lib/security/ directory:
cp -p /storage/alt_root/usr/lob/jvm/openjdk-java17-headless.x86_64/lib/security/cacerts /etc/alternatives/jre/lib/security/cacerts
Restart SDDC Manager services:
/opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

Additional Information

The symptoms are similar to: SDDC manager UI failed load due to empty "trusted_certificates.store"