vAPI Endpoint Service after a manual restart would later stop

Products

• VMware vCenter Server

Issue/Introduction

• The vAPI Endpoint Service on the vCenter Server after being manual started, would stop after sometime continuously.
• Manual restart the service will work for sometime and then it will stops again after sometime (hours later).
• Log signature looks like the below:
  endpoint.log
YYYY-MM-DDTHH:MM:SS | WARN  | vAPI-I/O dispatcher-0     | RequestDurationLogProvider     | Slow operation. Request to service: com.vmware.cis.tagging.tag_association, operation: list_attached_objects took 7382 ms
YYYY-MM-DDTHH:MM:SS | WARN  | jetty-default-40          | RequestRateLimitedProvider     | User name cannot be obtained.
YYYY-MM-DDTHH:MM:SS | WARN  | jetty-default-491         | RequestRateLimitedProvider     | User name cannot be obtained.
YYYY-MM-DDTHH:MM:SS | WARN  | state-manager1            | ApiConnectionsLsUtil           | Cannot find metadata source definitions in VAPI endpoint Service Endpoint of type com.vmware.api.pbm with protocol vapi.json.http at http://localhost:8190/pbm/vapi
YYYY-MM-DDTHH:MM:SS | WARN  | state-manager1            | ApiConnectionsLsUtil           | Unable to find metadata endpoint in service Service with localization key cis.sms.ServiceDescription and id 6f1bea2d-b282-4b12-8f45-fbe74dc6bab9.

Environment

VMware vCenter Server 7.x

Cause

The issue is a race condition between the envoy service and the rhttpproxy service.  

Resolution

In order to workaround this issue:

• • SSH to the VCSA using PuTTY or similar SSH client.
  • Enter the shell of the vCSA. (If the prompt shows command> that is the appliance shell. The below commands cannot be ran in appliance shell.).
  • Change directory to svcCfgfiles by running the command: cd /etc/vmware/vmware-vmon/svcCfgfiles/
  • Open the text editor of envoy.json by running the command: vi envoy.json
  • In the text editor window, press i on your keyboard and input the line: "DependsOn" : ["rhttpproxy"], as seen below:
    {
    "Name" : "envoy",
    "StartCommand" : "/usr/sbin/envoy",
    "StartCommandArgs" : [
    ],
    "RunAsUser" : "envoy",
    "ProcessFdLimit": 30000,
    "ApiHealthCommand": "%VMWARE_PYTHON_BIN%",
    "ApiHealthCommandArgs": [
        "/usr/lib/vmware-envoy/envoy-vmon-apihealth.py"
    ],
    "ApiHealthCmdTimeout": 120,
    "StartTimeout" : 35,
    "StopTimeout" : 15,
    "DependsOn" : ["rhttpproxy"],
    "StartupType" : "AUTOMATIC",
    "StreamRedirectFile": "/var/log/vmware/envoy/envoy_init.log",
    "RecoveryActionProfiles" :
    {
        "DEFAULT" :
        {
            "CRASH" : ["RESTART_SERVICE", "RESTART_SERVICE", "NO_ACTION"],
            "HEALTHFAIL" : ["RESTART_SERVICE", "RESTART_SERVICE", "NO_ACTION"]
        },
        "FAILOVER" :
        {
            "CRASH" : [
                "RESTART_SERVICE", "RESTART_SERVICE",
                "RESTART_SERVICE", "RESTART_SERVICE",
                "HAFAILOVER"
            ],
            "HEALTHFAIL" : [
                "RESTART_SERVICE", "RESTART_SERVICE",
                "RESTART_SERVICE", "RESTART_SERVICE",
  • Exit the window by pressing Esc along with wq!.
  • Then run the command: service-control --restart envoy