SDDC manager UI failed load due to corrupted cacerts file
search cancel

SDDC manager UI failed load due to corrupted cacerts file

book

Article ID: 397788

calendar_today

Updated On:

Products

VMware Cloud Foundation

Issue/Introduction

  • SDDC Manager UI is not loading
  • commonsvcs service fails to start
  • Error in var/log/vmware/vcf/sddc-manager-ui-app/sddcManagerServer.log   
YYYY-MM-DDTHH:MM:SS.844+0000 WARN [4394c04c2fd24b43, 5f03512217194914] [services/pscUtils.js, init-pscs, attemptPSCInitWithRetry:106]
100.109: VError: PSC Initilization attempt "12" failed: Failed to initiate PSC: Unable to initialize psc inventory data: Failed to fetch results from /inventory/pscs api: 502 - "<html>\n<head><title>502 Bad Gateway</title></head>\n<body>\n<center><h1>502 Bad Gateway</h1></center>\n<hr><center>nginx</center>\n</body>\n</html>\n"
    at Object.initializationPscError (/opt/vmware/vcf/sddc-manager-ui-app/server/src/errors/VCFError.js:100:5)
    at attemptPSCInitWithRetry (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/pscUtils.js:99:46)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Error Info: {"retryCount":12,"status":403,"errorModule":100,"errorCode":109}
  • Error in /var/log/vmware/vcf/commonsvcs/vcf-commonsvcs.log
YYYY-MM-DDTHH:MM:SS.087+0000 ERROR [common,0000000000000000,0000] [c.v.e.s.c.util.LocalProcessService,main] Local Command Failed with exit value 1.
Output Logs :
Error Logs are stored at LocalProcess ERROR: 2025-05-05 15:05:11 - /bin/cp: error writing '/usr/lib/jvm/openjdk-java17-headless.x86_64/lib/security/cacerts': No space left on device
YYYY-MM-DDTHH:MM:SS.622+0000 ERROR [common,0000000000000000,0000] [o.s.boot.SpringApplication,main] Application run failed
org.springframework.context.ApplicationContextException: Unable to start web server; nested exception is org.springframework.boot.web.server.WebServerException: Unable to start embedded Tomcat
instantiate [javax.net.ssl.TrustManager]: Factory method 'trustManager' threw exception; nested exception is com.vmware.vcf.secure.errors.VcfSecurityException: Failed to initialize trust manager
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:800)

Environment

SDDC 5.x

Cause

cacerts file corrupted due to out of space condition

lrwxrwxrwx 1 root root   64 Mon DD HH:MM cacerts.jks -> /usr/lib/jvm/openjdk-java17-headless.x86_64/lib/security/cacerts
-rw-r--r-- 1 root root 192K Mon DD HH:MM cacerts

Resolution

  1. Take a snapshot of the SDDC Manager VM
  2. SSH to SDDC Manager VM using vcf account and su to root
  3. Rebuild the cacerts file or copy one from a running system
  4. Restart SDDC Manager services
/opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

Additional Information

Powered by Wolken Software