VMware Secure Token Service not working

Products

• VMware vCenter Server

Issue/Introduction

When accessing vCenter UI you get this message No healthy upstream.

Login to the VAMI you see that the VMware Secure Token Service is not running.
Manual restart  of service is working but service stop working after some time

 

In vCenter in /var/log/vmware/vmon/vmon.log :

Wa(03) host-2765 <sts> Failed to write pid 1937942 to file. Error: No space left on device
Wa(03) host-2765 <sts> Failed to write pid 1937942 to file. Error: No space left on device
Wa(03)+ host-2765   File "/usr/lib/vmidentity/install/sts-vmon-health-checker.py", line 160, ]]

Wa(03) host-2765 <sts> Service exited. Exit code 120
Wa(03) host-2765 <sts> Service failed to recover. Try again. Fail count 4
In(05) host-2765 <sts-prestart> Constructed command: /usr/bin/python /usr/lib/vmidentity/install/sts-prestart-script.py /var/log/vmware/sso/sts-prestart.log
In(05) host-2765 <sts> Service pre-start command completed successfully.
In(05) host-2765 <sts> Constructed command: /usr/lib/vmidentity/install/sts-start-script.sh
Wa(03) host-2765 <sts> Service exited. Exit code 120
Er(02) host-2765 <sts> Service reached max quick failure count. Give up!!!

 

Checking space usage on VCSA using df -h you see similar output

(space usage will vary between environments)

 

Filesystem                                   1K-blocks      Used  Available Use% Mounted on
devtmpfs                                          4096         0       4096   0% /dev
tmpfs                                         29819644      2292   29817352   1% /dev/shm
tmpfs                                         11927860      1348   11926512   1% /run
tmpfs                                             4096         0       4096   0% /sys/fs/cgroup
/dev/mapper/vg_root_0-lv_root_0               49222292  19977688   26711844  43% /
/dev/sda3                                       498900     37428     424776   9% /boot
tmpfs                                         29819648      4900   29814748   1% /tmp
/dev/sda2                                        10202      1978       8224  20% /boot/efi
/dev/mapper/vg_lvm_snapshot-lv_lvm_snapshot 1030987928        28  978542924   1% /storage/lvm_snapshot
/dev/mapper/db_vg-db                          51282400   3352864   45292124   7% /storage/db
/dev/mapper/lifecycle_vg-lifecycle           102618040   3961596   93397592   5% /storage/lifecycle
/dev/mapper/dblog_vg-dblog                    25618660   2965552   21326416  13% /storage/dblog
/dev/mapper/vtsdblog_vg-vtsdblog              25618660     32804   24259164   1% /storage/vtsdblog
/dev/mapper/log_vg-log                        25618660  22443040    1848928  93% /storage/log    
/dev/mapper/netdump_vg-netdump                10210580        24    9670296   1% /storage/netdump
/dev/mapper/autodeploy_vg-autodeploy          25618660        40   24291928   1% /storage/autodeploy
/dev/mapper/archive_vg-archive               205305832 184772116   10031984  95% /storage/archive
/dev/mapper/core_vg-core                     102618040  68137136   29222052  70% /storage/core
/dev/mapper/imagebuilder_vg-imagebuilder      25618660        36   24291932   1% /storage/imagebuilder
/dev/mapper/updatemgr_vg-updatemgr           102618040   7156960   90202228   8% /storage/updatemgr
/dev/mapper/seat_vg-seat                    1474794768  12839344 1386966268   1% /storage/seat
/dev/mapper/vtsdb_vg-vtsdb                  1474794768     45328 1399760284   1% /storage/vtsdb

 

 

 

Environment

VMware vCenter Server 8.x

Cause

Due to lack of space on /storage/log partition service cannot work as expected.

Resolution

Apply steps from this article to identify what is consuming space and clear it 

vCenter log disk exhaustion or /storage/log full