502 Bad Gateway error when triggering a manual backup from the SDDC UI on SDDC 5.2.X / 5.X
Article ID: 432438
Updated On:
Products
Issue/Introduction
Symptoms:
Receive a 502 Bad Gateway error when triggering a manual backup from the SDDC UI.
Similar scenario as KB https://knowledge.broadcom.com/external/article?articleNumber=327193
Service has same behavior, as in the aforementioned KB
# curl localhost/sos/about
<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx</center>
</body>
</html>
However the error is slightly different
# journalctl --unit=sosrest --since '1 hour ago'
-- Journal begins at Mon 2026-02-23 04:35:00 UTC, ends at Mon 2026-03-09 14:22:49 UTC. --
<Month> <Day> <Time HH:MM:SS> 13:23:15 <SDDC manager FQDN> systemd[1]: sosrest.service: Scheduled restart job, restart counter is at 16318.
<Month> <Day> <Time HH:MM:SS> 13:23:15 <SDDC manager FQDN> systemd[1]: Stopped VMware Cloud Foundation Supportability and Serviceability(SoS) Service.
<Month> <Day> <Time HH:MM:SS> 13:23:15 <SDDC manager FQDN> systemd[1]: Started VMware Cloud Foundation Supportability and Serviceability(SoS) Service.
<Month> <Day> <Time HH:MM:SS> 13:23:16 <SDDC manager FQDN> sosrest[2569220]: Traceback (most recent call last):
<Month> <Day> <Time HH:MM:SS> 13:23:16 <SDDC manager FQDN> sosrest[2569220]: File "services/sosrest.py", line 182, in <module>
<Month> <Day> <Time HH:MM:SS> 13:23:16 <SDDC manager FQDN> sosrest[2569220]: File "services/sosrest.py", line 59, in __init__
<Month> <Day> <Time HH:MM:SS> 13:23:16 <SDDC manager FQDN> sosrest[2569220]: File "utils/util.py", line 96, in __init__
<Month> <Day> <Time HH:MM:SS> 13:23:16 <SDDC manager FQDN> sosrest[2569220]: File "utils/cache.py", line 30, in __init__
<Month> <Day> <Time HH:MM:SS> 13:23:16 <SDDC manager FQDN> sosrest[2569220]: File "/opt/vmware/sddc-support/services/../framework/../dependency/diskcache/core.py", line 450, in __init__
<Month> <Day> <Time HH:MM:SS> 13:23:16 <SDDC manager FQDN> sosrest[2569220]: raise EnvironmentError(
<Month> <Day> <Time HH:MM:SS> 13:23:16 <SDDC manager FQDN> sosrest[2569220]: PermissionError: [Errno 13] Cache directory "/opt/vmware/vcf/sddc-support//__soscache__" does not exist and could not be created
<Month> <Day> <Time HH:MM:SS> 13:23:16 <SDDC manager FQDN> systemd[1]: sosrest.service: Main process exited, code=exited, status=1/FAILURE
<Month> <Day> <Time HH:MM:SS> 13:23:16 <SDDC manager FQDN> systemd[1]: sosrest.service: Failed with result 'exit-code'.
<Month> <Day> <Time HH:MM:SS> 13:23:46 <SDDC manager FQDN> systemd[1]: sosrest.service: Scheduled restart job, restart counter is at 16319.Error complains about Permisison issue<DATE && TIME> <SDDC manager FQDN> sosrest[2569220]: PermissionError: [Errno 13] Cache directory "/opt/vmware/vcf/sddc-support//__soscache__" does not exist and could not be created
Environment
SDDC 5.2.X / 5.X
Cause
Path points to incorrect permissions
<DATE && TIME> <SDDC manager FQDN> sosrest[2569220]: PermissionError: [Errno 13] Cache directory "/opt/vmware/vcf/sddc-support//__soscache__" does not exist and could not be created
ls -althr
total 124K
drwx------ 2 root root 16K Apr 17 2025 lost+found
drwxr-xr-x 9 root root 4.0K Apr 17 2025 commonsvcs
-rwxr-xr-x 1 root root 23 Apr 17 2025 version.txt
-rw-r--r-- 1 root root 845 Apr 17 2025 java-systemwide-java-options.conf
-rw-r--r-- 1 root root 57K Apr 17 2025 java.default.security
drwxr-xr-x 2 root root 4.0K Apr 17 2025 fips
drwxr-xr-x 2 root root 4.0K Apr 17 2025 OSL
drwxr-xr-x 11 root root 4.0K Apr 17 2025 .
drwxr-xr-x. 10 root root 4.0K Jun 16 2025 ..
drwxr-x--- 3 root root 4.0K Jan 6 19:03 sddc-support <= root : root
drwxr-x--- 4 vcf_lcm vcf 4.0K Jan 6 19:10 lcm
drwxr-x--x 5 vcf_sddc_manager_ui_app vcf 4.0K Jan 6 19:16 sddc-manager-ui-app
drwxr-xr-x 5 vcf_domainmanager vcf 4.0K Jan 26 15:54 domainmanager
drwxr-xr-x 6 root root 4.0K Jan 26 15:55 operationsmanager
Resolution
1. Snapshot SDDC VM
2. Change permission to vcf_sos:vcf# chown -R vcf_sos:vcf sddc-support3. Restart sosrest service# systemctl restart sosrest.service
Feedback
