SDDC Manager sosrest service is down and it is not possible to start it manually

Products

• VMware SDDC Manager

Issue/Introduction

• SDDC Manager sosrest service is down and it is not possible to start it manually:

root@XXXXXXXXXX [ /opt/vmware/vcf/sddc-support ]# systemctl status sosrest.service -l

sosrest.service - VMware Cloud Foundation Supportability and Serviceability(sos) Services

Loaded: loaded (/etc/systemd/system/sosrest.service; enabled; vendor preset: enabled)

Active: activating (running) (auto-restart) (Result: exit-code) since Fri YYYY-MM-DDTHH:MM:SS; 5s ago

Process: 95416 ExecStart=/opt/vmware/sddc-support/sosrts (code=exited, status=1/FAILURE)

Main PID: 95416 (code=exited, status=1/FAILURE)

• /var/log/vmware/vcf/sddc-support/vcf-sos.log:

YYYY-MM-DDTHH:MM:SS INFO [vcf_bos] [db.spi:v.return_s.session:1785:MainThread] Closing session
YYYY-MM-DDTHH:MM:SS INFO [vcf_bos] [authProviders.py:handle_csam_recovery:129:MainThread] Clearing cache contents
YYYY-MM-DDTHH:MM:SS INFO [vcf_bos] [cache.c:clear_cache:144:MainThread] Clearing cache enabled
YYYY-MM-DDTHH:MM:SS INFO [util.py:read_and_set_config_defaults:3761:MainThread] Reading config default values from /opt/vmware/addc-support/conf/vos-app.conf
YYYY-MM-DDTHH:MM:SS INFO [util.py:read_and_set_config_defaults:3761:MainThread] Compiled reading default values
YYYY-MM-DDTHH:MM:SS INFO [util.py:read_and_set_config_defaults:3774:MainThread] Config default values {seconds} esm_timeout=3600, max_timeout=7200, vcenter_timeout=3600, wcp_timeout=3600, min_timeout=3600, health_check_timeout=3600, process_count=5
YYYY-MM-DDTHH:MM:SS INFO [util.py:handle_default_values:3824:MainThread] Log collection timeout is: 7800
YYYY-MM-DDTHH:MM:SS INFO [util.py:handle_default_values:3824:MainThread] Setting esm and vcd port with user and vcd
YYYY-MM-DDTHH:MM:SS ERROR [oscar.py:startup:125:MainThread] [Errno 13] Permission denied: '/usr/lib/python3.10/site-packages/requests_toolbelt-1.0.0.dist-info'
YYYY-MM-DDTHH:MM:SS INFO [oscar.py:startup:125:MainThread] SOS REST service failed while startup of Guncore Server

 

• Rebooting the SDDC Manager allows the service to come up but it crashes again within a minute.

Cause

This issue is caused by incorrect permissions set on a required directory.

In this case the directory in question is: /usr/lib/python3.10/site-packages/requests_toolbelt-1.0.0.dist-info

Resolution

Assign required permissions recursively to the directory noted in the logs.

e.g.

chmod -R 755 /usr/lib/python3.10/site-packages/requests_toolbelt-1.0.0.dist-info

Start the sosrest service:

systemctl restart sosrest.service