vCenter services fail to start due to missing Log directories

Products

• VMware vCenter Server

Issue/Introduction

Many Services may fail to start due to missing respective Log directories.

 

Incorrect ownership for topologysvc service, which is evident under /var/log/vmware/topologysvc/topology-svcs.log:

YYYY-MM-DDTHH-MM-SS Er(02) host-xxxx Failed to check existence of file: /var/log/vmware/topologysvc/topologysvc-runtime.log.stdout. Error: Permission denied
YYYY-MM-DDTHH-MM-SS Er(02) host-xxxx [Child 19722] Failed to fetch a file name for stream redirection.
YYYY-MM-DDTHH-MM-SS Wa(03) host-xxxx <topologysvc> Service exited. Exit code 254

drwx------ 2 root            root                4096 MMM DD HH:MM topologysvc  (Incorrect permission)

drwx------ 2 topologysvc     root                4096 MMM DD HH:MM topologysvc  (Correct permission)

Missing directory for rhttpproxy Service, which is evident under /var/log/vmware/rhttpproxy/rhttpproxy.log:

YYYY-MM-DDTHH-MM-SS In(05) host-xxxx Received start request for rhttpproxy
YYYY-MM-DDTHH-MM-SS In(05) host-xxxx <rhttpproxy-prestart> Constructed command: /usr/bin/banner_ctrl set --http_ep_service rhttpproxy
YYYY-MM-DDTHH-MM-SS In(05) host-xxxx <rhttpproxy> Service pre-start command completed successfully.
YYYY-MM-DDTHH-MM-SS In(05) host-xxxx <rhttpproxy> Constructed command: /usr/sbin/rhttpproxy -r /etc/vmware-rhttpproxy/config.xml -d /etc/vmware-rhttpproxy/endpoints.conf.d -f /etc/vmware-rhttpproxy/endpoints.conf.d/vpxd.conf
YYYY-MM-DDTHH-MM-SS Er(02) host-xxxx <rhttpproxy> [Process: 23229] Failed to redirect std-stream. Error: No such file or directory
YYYY-MM-DDTHH-MM-SS Wa(03) host-xxxx <rhttpproxy> Service exited. Exit code 254
YYYY-MM-DDTHH-MM-SS In(05) host-xxxx <rhttpproxy-post-stop> Constructed command: /usr/bin/banner_ctrl set --http_ep_service lighttp
YYYY-MM-DDTHH-MM-SS Wa(03) host-xxxx [ReadSvcSubStartupData] No startup information from rhttpproxy.

The same issue can be observed for wcp Service, which is evident under /var/log/vmware/wcp/wcpsvc.log:

YYYY-MM-DDTHH-MM-SS fatal wcp [crypto/cryptography.go:xx] Cannot load secret key from file: open /etc/vmware/wcp/keyForCryptography.dat: permission denied
YYYY-MM-DDTHH-MM-SS error wcp [crypto/cryptography.go:xx] Unable to read key from file /etc/vmware/wcp/keyForCryptography.dat. Error: open /etc/vmware/wcp/keyForCryptography.dat: permission denied

 

Environment

vCenter Server 7.x

vCenter Server 8.x

Cause

User might have accidentally deleted the log directories while performing the log filesystem cleanup.

Review the cause of the service start failure in vMon logs.

Resolution

Change the ownership of the log directory as required.

Example:

chown wcp /var/log/vmware/wcp

Additional Information

Note: If the directory is not found, recreate the directory with the same name and change the ownership.

If the directory exists, but the service logs are owned by root, a grep command such as the following can be run against vmon to check  for "Incorrect Permissions" on the service failing to start:

grep '<topologysvcs>' /var/log/vmware/vmon/vmon.log | grep -ivE 'health|counter'