ERROR: "service name "wcp" is invalid" and issues putting host into maintenance mode.

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Hosts cannot enter Maintenance Mode:
  
  
• Failed to enter namespaces maintenance mode due to Error: system_error Messages: vapi.send.failed<Send of frame failed: N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.)


• Workload Management UI will not be available with output: "None of the hosts connected to this vCenter are licensed for Workload Management."



• /var/log/vmware/vmon/vmon.log will have an entry similar to the following:

2024-06-07T18:04:11.554Z Er (02) host-#### Invalid Json formatted file /etc/vmware/vmware-vmon/svcCfgfiles/wcpsvc-vmonconfig.json

There are also reports of failed backups and restores of VM's from 3rd party backup solutions.

Cause

There is an issue with the wcpsvc-vmonconfig.json. The issues could range from a line commented out to the opening bracket missing:

Example 1:

A line is commented out with # in /etc/vmware/vmware-vmon/svcCfgfiles/wcpsvc-vmonconfig.json

How it looks:

# "PreStartCommandArgs" : ["/usr/lib/vmware-wcp/wcpsvc-prestart.py"]

It should be:

"PreStartCommandArgs" : ["/usr/lib/vmware-wcp/wcpsvc-prestart.py"]

Example 2:

The opening bracket for the file is missing:

How it looks:

  "Name" : "wcp"

It should be:

{
    "Name" : "wcp"

Resolution

Edit the wcpsvc-vmonconfig.json according to the error, so that the formatting is correct. Refer to the "Additional Information" section for an example of the wcpsvc-vmonconfig.json has been provided below for reference

Restart all services so vMon can see the change:

1.  SSH into the vCenter using the root credential
2.  Please run the following command: service-control --stop --all && service-control --start --all

Additional Information

This can also occur if the file has been changed in any way. A missing comma, for instance, can cause the wcp service to become "invalid". Here is an example of what the file should look like in 7.0 U3:

{
    "Name" : "wcp",
    "StartCommand" : "/usr/lib/vmware-wcp/wcpsvc.launcher",
    "StartCommandArgs": ["--port", "8920", "--logfile", "/var/log/vmware/wcp/wcpsvc.log",
                         "--configfile", "/etc/vmware/wcp/wcpsvc.yaml",
                         "--audit-logfile", "/var/log/vmware/wcp/wcp-audit.log",
                         "--incident-logfile", "/var/log/vmware/wcp/wcp-incident.log"],
    "PreStartCommand" : "%VMWARE_PYTHON_BIN%",
    "PreStartCommandArgs" : ["/usr/lib/vmware-wcp/wcpsvc-prestart.py"],
    "StreamRedirectFile" : "/var/log/vmware/wcp/stdstream.log",
    "RecoveryActionProfiles" :
    {
        "DEFAULT" :
        {
            "CRASH" : ["RESTART_SERVICE", "RESTART_SERVICE", "RESTART_SERVICE", "RESTART_SERVICE", "RESTART_SERVICE", "NO_ACTION"],
            "HEALTHFAIL" : ["NO_ACTION"]
        }
    },
    "WorkingDir" : "/usr/lib/vmware-wcp/",
    "DependsOn" : ["vpxd", "vpxd-svcs", "eam", "vmware-vpostgres", "lookupsvc", "cis-license", "vapi-endpoint", "sts", "trustmanagement"],
    "ApiHealthCommand" : "%VMWARE_PYTHON_BIN%",
    "ApiHealthCommandArgs" : ["/usr/lib/vmware-vmon/vmonApiHealthCmd.py", "-n", "wcp", "-u", "/wcp/health", "-t", "10"],
    "StartTimeout" : 120,
    "StopTimeout" : 30
}