vsan-health service fails to start | FileRotateByRename failed: Permission denied

Products

• VMware vCenter Server 8.0
• VMware vCenter Server 7.0

Issue/Introduction

vsan-health service fails to start

service-control : 

root@vcsa [ /var/log/vmware ]# service-control --start vsan-health
Operation not cancellable. Please wait for it to finish...
Performing start operation on service vsan-health...
Error executing start on service vsan-health. Details {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "vsan-health"
            ],
            "localized": "An error occurred while starting service 'vsan-health'"

        }
    ],

    "componentKey": null,
    "problemId": null,
    "resolution": null
}
Service-control failed. Error: {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "vsan-health"
            ],            "localized": "An error occurred while starting service 'vsan-health'"

        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
}

vmon.log :

XXXX-XX-XXTXX:XX:42.047Z In(05) host-17519 <vsan-health> Constructed command: /usr/sbin/vsanvcmgmtd -s -c /usr/lib/vmware-vpx/vsan-health/VsanVcMgmtConfig.xml -u /etc/vmware-vsan-health/VsanMgmtCustomizedConfig.xml
XXXX-XX-XXTXX:XX:42.048Z In(05) host-17519 FILE: FileRotateByRename: failed to remove /var/log/vmware/vsan-health/vmware-vsan-health-runtime.log-4.stdout: Permission denied
XXXX-XX-XXTXX:XX:42.048Z In(05) host-17519 FILE: /var/log/vmware/vsan-health/vmware-vsan-health-runtime.log-3.stdout: rename of /var/log/vmware/vsan-health/vmware-vsan-health-runtime.log-4.stdout -> FileRotateByRename failed: Permission denied
XXXX-XX-XXTXX:XX:42.048Z In(05) host-17519 FILE: /var/log/vmware/vsan-health/vmware-vsan-health-runtime.log-2.stdout: rename of /var/log/vmware/vsan-health/vmware-vsan-health-runtime.log-3.stdout -> FileRotateByRename failed: Permission denied
XXXX-XX-XXTXX:XX:42.048Z In(05) host-17519 FILE: /var/log/vmware/vsan-health/vmware-vsan-health-runtime.log-1.stdout: rename of /var/log/vmware/vsan-health/vmware-vsan-health-runtime.log-2.stdout -> FileRotateByRename failed: Permission denied
XXXX-XX-XXTXX:XX:42.048Z In(05) host-17519 FILE: /var/log/vmware/vsan-health/vmware-vsan-health-runtime.log-0.stdout: rename of /var/log/vmware/vsan-health/vmware-vsan-health-runtime.log-1.stdout -> FileRotateByRename failed: Permission denied
XXXX-XX-XXTXX:XX:42.048Z In(05) host-17519 FILE: /var/log/vmware/vsan-health/vmware-vsan-health-runtime.log.stdout: rename of /var/log/vmware/vsan-health/vmware-vsan-health-runtime.log-0.stdout -> FileRotateByRename failed: Permission denied
XXXX-XX-XXTXX:XX:42.048Z In(05) host-17519 FILE: FileRotateByRename: failed to remove /var/log/vmware/vsan-health/vmware-vsan-health-runtime.log-4.stderr: Permission denied
XXXX-XX-XXTXX:XX:42.048Z In(05) host-17519 FILE: /var/log/vmware/vsan-health/vmware-vsan-health-runtime.log-3.stderr: rename of /var/log/vmware/vsan-health/vmware-vsan-health-runtime.log-4.stderr -> FileRotateByRename failed: Permission denied
XXXX-XX-XXTXX:XX:42.048Z In(05) host-17519 FILE: /var/log/vmware/vsan-health/vmware-vsan-health-runtime.log-2.stderr: rename of /var/log/vmware/vsan-health/vmware-vsan-health-runtime.log-3.stderr -> FileRotateByRename failed: Permission denied
XXXX-XX-XXTXX:XX:42.048Z In(05) host-17519 FILE: /var/log/vmware/vsan-health/vmware-vsan-health-runtime.log-1.stderr: rename of /var/log/vmware/vsan-health/vmware-vsan-health-runtime.log-2.stderr -> FileRotateByRename failed: Permission denied
XXXX-XX-XXTXX:XX:42.048Z In(05) host-17519 FILE: /var/log/vmware/vsan-health/vmware-vsan-health-runtime.log-0.stderr: rename of /var/log/vmware/vsan-health/vmware-vsan-health-runtime.log-1.stderr -> FileRotateByRename failed: Permission denied
XXXX-XX-XXTXX:XX:42.048Z In(05) host-17519 FILE: /var/log/vmware/vsan-health/vmware-vsan-health-runtime.log.stderr: rename of /var/log/vmware/vsan-health/vmware-vsan-health-runtime.log-0.stderr -> FileRotateByRename failed: Permission denied

Cause

This issue happens when the permissions to vsan-health folder under /var/log/vmware is not correct or there will be invalid symlink created

Eg :Below image shows incorrect permissions set to vsan-health folder and the folder is symlinked to /storage/lifecycle/vsan-health

Resolution

To fix this issue, please follow the steps below :


Take snapshot of the vCenter appliance. 

cd /storage/lifecycle

 -> We will copy the contents to a different folder.

 cp -r vsan-health/ /storage/core/

 cd /var/log/vmware

 rm -rf vsan-health

 mkdir vsan-health

 
-> Copy the data back

 cp -r /storage/core/vsan-health/* /var/log/vmware/vsan-health/

 -> Will assign the proper folder and file level permissions.

 chown vsan-health:root vsan-health/

 
cd vsan-health/

chown -R vsan-health:users *

 

Once done, start vsan-health service by service-control --start vsan-health