Unable to start services using service-control on vCenter Server 7.x.

Products

• VMware vCenter Server

Issue/Introduction

Symptoms

• The services on vCenter Server cannot be started using either service-control or vmon-cli.
• When running service-control, the command fails with an error like the following:
Performing start operation on service analytics...
Error executing start on service analytics. Details {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "analytics"
            ],
            "localized": "An error occurred while starting service 'analytics'"
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
}
...
  Note: In this case, the analytics service failed to start; however, in the scenario described in this article, other services besides analytics may also fail to start.
• The log file /var/log/vmware/vmon/vmon.log contains entries showing "Too many open files." In this example, these logs appeared when executing service-control --start analytics.
  YYYY-MM-DDTHH:mm:SS.SSSZ| host-1818| I005: Received start request for analytics
YYYY-MM-DDTHH:mm:SS.SSSZ| host-1818| E002: <analytics-healthcmd> stderr pipe creation failed. Too many open files

• The vmon process has opened a large number of vMonTelemetry.db files, as shown in the following command:
  root@vcsa [ ~ ]# lsof -p `pgrep -x vmon` | grep -c vMonTelemetry.db
1013
  Note: In this example, vmon holds 1013 file descriptors for vMonTelemetry.db, but this number may vary depending on the environment.

Environment

vCenter Server 7.x

Cause

The root cause is a file descriptor leak issue in vmon service. Each execution of service-controll --restart [SERVICE] increments the file descriptor count for vMonTelemetry.db by two. If the service is restarted repeatedly, the vmon process will eventually exhaust its file descriptors.

Resolution

This is fixed in vCenter Server 7.0 U3c and later versions. If this issue occurs, a restart of the vCenter Server is required for recovery.