502 Bad Gateway error when triggering a manual backup from the SDDC UI
search cancel

502 Bad Gateway error when triggering a manual backup from the SDDC UI

book

Article ID: 327193

calendar_today

Updated On: 02-19-2025

Products

VMware Cloud Foundation

Issue/Introduction

Symptoms:

Receive a 502 Bad Gateway error when triggering a manual backup from the SDDC UI. image.png

High number of operationsmanager.logs in the operationsmanager directory.  
ls -lh /var/log/vmware/vcf/operationsmanager/operationsmanager* | wc -l
1547
sosrest service is in a failed state.  
curl localhost/sos/about
<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx</center>
</body>
</html>


Environment

VMware Cloud Foundation 4.x

Cause

1.During a SOS log collection a backup was triggered. 
2.Log collection operation is aborted abnormally and the temporary bundle is not cleaned up. 
Expected files in the sddc-support directory. 
ls -lh /opt/vmware/vcf/sddc-support/
total 1.5M
drwxr-xr-x 2 root root 4.0K Jun 15 19:36 __soscache__
-rw-r--r-- 1 root root  20K Jun 29  2021 sos.db
-rw-r--r-- 1 root root  20K Sep 23  2021 soservice.db
-rw-r--r-- 1 root root 1.4M May 10 05:00 status.json

Resolution

Issue will be addressed in a future release of SDDC.

Workaround:
If any symlink or invalid files are in the backup directory path it should be manually deleted. 

0.Take a snapshot of the SDDC Manager.
1.Identify the invalid file/symlink from the journalctl log.

journalctl --unit=sosrest --since '1 hour ago'

2.Search for similar errors to "Failed with result 'exit-code'." or "FileNotFoundError"
example:

journalctl --unit=sosrest --since '1 hour ago'
==============================================================
Jun 01 02:17:54 sddc-manager.vcf.corp.local sosrest[1789]: FileNotFoundError: [Errno 2] No such file or directory: '/var/log/vmware/vcf/sddc-support/backup-2023-05-08-16-22-45-1857/vcf-backup-sddc-manager.vcf.corp.local-2023-05-08-16-22-45/configuration/opt/vmware/vcf/sddc-support/sddc-202303291541/var/log/dracut.log'

3.Delete the invalid file from SDDC.

rm /var/log/vmware/vcf/sddc-support/backup-2023-05-08-16-22-45-1857/vcf-backup-sddc-manager.vcf.corp.local-2023-05-08-16-22-45/configuration/opt/vmware/vcf/sddc-support/sddc-202303291541/var/log/dracut.log

4.Restart the sosrest service 

systemctl restart sosrest.service

5.Verify the sosrest service is still active

systemctl status sosrest.service -l 

6.Cleanup the excessive operationsmanager logs from the operationsmanager directory.

Powered by Wolken Software