SDDC UI fails to load with error "Failed to initiate PSC: Unable to initialize psc inventory data: Failed to fetch results from /inventory/pscs api: Request failed with status code 502"

Products

VMware SDDC Manager

Issue/Introduction

SDDC UI fails to load due to root partition getting 100% full.
This in turn causes the cacert corruption.
/var/log/vmware/vcf/sddc-manager-ui-app/sddcManagerServer.log

VError: Failed to initiate PSC: Unable to initialize psc inventory data: Failed to fetch results from /inventory/pscs api: Request failed with status code 502
at Object.initiatePscError (/opt/vmware/vcf/sddc-manager-ui-app/server/src/errors/VCFError.js:104:5)
at attemptPSCInit (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/pscUtils.js:72:26)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async attemptPSCInitWithRetry (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/pscUtils.js:99:17)
Error Info: {"errorModule":100,"errorCode":108,"code":"ERR_BAD_RESPONSE"}
caused by:
100.106: VError: Unable to initialize psc inventory data: Failed to fetch results from /inventory/pscs api: Request failed with status code 502
at Object.initializePscInventoryError (/opt/vmware/vcf/sddc-manager-ui-app/server/src/errors/VCFError.js:104:5)
at retrieveIdpPscDetails (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/pscUtils.js:43:26)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async attemptPSCInit (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/pscUtils.js:57:26)
at async attemptPSCInitWithRetry (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/pscUtils.js:99:17)
Error Info: {"errorModule":100,"errorCode":106,"code":"ERR_BAD_RESPONSE"}
caused by:
100.146: VError: Failed to fetch results from /inventory/pscs api: Request failed with status code 502
at Object.failedToFetchResultsFromInventoryError (/opt/vmware/vcf/sddc-manager-ui-app/server/src/errors/VCFError.js:104:5)
at retrieveIDPPsc (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/credentialRetriever.js:61:26)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async retrieveIdpPscDetails (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/pscUtils.js:29:26)
at async attemptPSCInit (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/pscUtils.js:57:26)
at async attemptPSCInitWithRetry (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/pscUtils.js:99:17)
Error Info: {"errorModule":100,"errorCode":146,"code":"ERR_BAD_RESPONSE"}

All the service logs such as domainmanager, operationsmanager, commomsvc etc indicate a problem accessing the trust store.

Caused by: java.security.KeyStoreException: problem accessing trust store
at java.base/sun.security.ssl.TrustManagerFactoryImpl.engineInit(TrustManagerFactoryImpl.java:73)
at java.base/javax.net.ssl.TrustManagerFactory.init(TrustManagerFactory.java:282)
at okhttp3.internal.platform.Platform.platformTrustManager(Platform.kt:80)
at okhttp3.OkHttpClient.<init>(OkHttpClient.kt:237)
at okhttp3.OkHttpClient.<init>(OkHttpClient.kt:222)
at com.vmware.cloud.foundation.rest.commonsvcs.runtime.ApiClient.<init>(ApiClient.java:77)
at com.vmware.cloud.foundation.rest.commonsvcs.runtime.Configuration.<clinit>(Configuration.java:17)
... 222 common frames omitted
Caused by: java.io.EOFException: null

Following SDDC Manager UI becomes inaccessible due to root (/dev/sda4) partition filling up to clean up the root partition does not stay consistent, the partition gets filled within 2-3 days.
Root partition gets filled up due to excess logging on the messages log file. The size of the messages log file is huge as well.

Environment

VMware Cloud Foundation 5.2.1

Cause

SOS DB corruption.

Consistent queries trying to fetch a record for a particular ID that is no more available is seen in messages.log file

/var/log/messages/

YYYY-MM-DDTHH:MM:SS sosrest[2021] File "framework/dbinterface/db_api.py", line 273, in fetch_a_record
YYYY-MM-DDTHH:MM:SS sosrest[2021] Message: 'No record found for query'
YYYY-MM-DDTHH:MM:SS sosrest[2021] Arguments: ()
YYYY-MM-DDTHH:MM:SS sosrest[2018] --- Logging error ---
YYYY-MM-DDTHH:MM:SS sosrest[2018] Traceback (most recent call last):
YYYY-MM-DDTHH:MM:SS sosrest[2018] File "/usr/lib/python3.10/logging/handlers.py", line 74, in emit
YYYY-MM-DDTHH:MM:SS sosrest[2018] self.doRollover()
YYYY-MM-DDTHH:MM:SS sosrest[2018] File "utils/log.py", line 366, in doRollover
YYYY-MM-DDTHH:MM:SS sosrest[2018] File "/usr/lib/python3.10/logging/handlers.py", line 435, in doRollover
YYYY-MM-DDTHH:MM:SS sosrest[2018] self.rotate(self.baseFilename, dfn)
YYYY-MM-DDTHH:MM:SS sosrest[2018] File "utils/log.py", line 377, in rotate
YYYY-MM-DDTHH:MM:SS sosrest[2018] FileNotFoundError: [Errno 2] No such file or directory: '/var/log/vmware/vcf/sddc-support/vcf-sos-svcs.log' -> '/var/log/vmware/vcf/sddc-support/vcf-sos-svcs.log.YYYY-MM-DD'
YYYY-MM-DDTHH:MM:SS sosrest[2018] Call stack:
YYYY-MM-DDTHH:MM:SS sosrest[2021] 2025-06-04 05:09:02 [2021] [INFO] No record found for query
YYYY-MM-DDTHH:MM:SS sosrest[2019] --- Logging error ---
YYYY-MM-DDTHH:MM:SS sosrest[2019] Traceback (most recent call last):
YYYY-MM-DDTHH:MM:SS sosrest[2019] File "framework/dbinterface/db_api.py", line 64, in return_a_session
YYYY-MM-DDTHH:MM:SS sosrest[2019] File "framework/dbinterface/db_api.py", line 268, in fetch_a_record
YYYY-MM-DDTHH:MM:SS sosrest[2018] Message: "Fetching records for table <class 'framework.dbinterface.database.Tasks'> with id in ['########-17f0-4692-82c7-###########']"
YYYY-MM-DDTHH:MM:SS sosrest[2018] Arguments: ()
YYYY-MM-DDTHH:MM:SS sosrest[2018] 2025-06-04 05:09:02 [2018] [INFO] Fetching records for table <class 'framework.dbinterface.database.Tasks'> with id in ['########-17f0-4692-82c7-###########']
YYYY-MM-DDTHH:MM:SS sosrest[2021] --- Logging error ---
YYYY-MM-DDTHH:MM:SS sosrest[2021] Traceback (most recent call last):
YYYY-MM-DDTHH:MM:SS sosrest[2021] File "/usr/lib/python3.10/logging/handlers.py", line 74, in emit
YYYY-MM-DDTHH:MM:SS sosrest[2021] self.doRollover()
YYYY-MM-DDTHH:MM:SS sosrest[2021] File "utils/log.py", line 366, in doRollover
YYYY-MM-DDTHH:MM:SS sosrest[2021] File "/usr/lib/python3.10/logging/handlers.py", line 435, in doRollover
YYYY-MM-DDTHH:MM:SS sosrest[2021] self.rotate(self.baseFilename, dfn)
YYYY-MM-DDTHH:MM:SS sosrest[2021] File "utils/log.py", line 377, in rotate
YYYY-MM-DDTHH:MM:SS sosrest[2021] FileNotFoundError: [Errno 2] No such file or directory: '/var/log/vmware/vcf/sddc-support/vcf-sos-svcs.log' -> '/var/log/vmware/vcf/sddc-support/vcf-sos-svcs.log.YYYY-MM-DD'

Resolution

Note: This will remove the backup history from SDDC Manager

This requires changes in the SDDC manager database. Kindly ensure to take a snapshot of the SDDC Manager before making the changes

SSH into the SDDC Manager Appliance with the vcf user and then elevate to root with su
Stop the sosrest service using systemctl stop sosrest.service

Move the corrupted files out of the sddc-support directory using below commands

mv /opt/vmware/vcf/sddc-support/soservice.db /home/vcf

mv /opt/vmware/vcf/sddc-support/status.json /home/vcf

mv /opt/vmware/vcf/sddc-support/.status-tmp.json /home/vcf

Remove the sos service tasks from Task Aggregator using the below commands

/usr/pgsql/13/bin/psql -h localhost -U postgres -d platform -c "delete from task_metadata where task_url like '%sos%'"

/usr/pgsql/13/bin/psql -h localhost -U postgres -d platform -c "delete from task_metadata where task_type='SDDCMANAGER_BACKUP'"

Cycle the SDDC services using

/opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

Verify the UI is working as expected and the "Fetching Task" entries have resolved.
The root partition should stop growing abnormally.

Additional Information

If the above resolution does not work, please rebuild the cacerts file or copy one from a running system.

Kindly follow the steps in the KB article: https://knowledge.broadcom.com/external/article?articleNumber=397788