Error: "com.vmware.vapi.std.errors.unauthenticated" and "vapi.security.authentication.invalid" for the WCP service causing multiple workflow failures
search cancel

Error: "com.vmware.vapi.std.errors.unauthenticated" and "vapi.security.authentication.invalid" for the WCP service causing multiple workflow failures

book

Article ID: 322225

calendar_today

Updated On:

Products

VMware Cloud Foundation VMware vCenter Server

Issue/Introduction

  • Unable to put a host in maintenance mode

  • In the vCenter Server /var/log/vmware/vpxd/vpxd.log the following entries occur,

    YYYY-MM-DDTHH:MM:SS.###-##:## info vpxd[10034] [Originator@6876 sub=MoHost opID=opId-18b14-105289-d9] WCP exitMaintenanceMode vAPI returns error: Error:
    -->    com.vmware.vapi.std.errors.unauthenticated
    --> Messages:
    -->    vapi.security.authentication.invalid<Unable to authenticate user>
    -->   YYYY-MM-DDTHH:MM:SS.###-##:## error vpxd[10034] [Originator@6876 sub=MoHost opID=opId-18b14-105289-d9] [Delete] Failed to delete vAPI session. Error:
    --> Error:
    -->    com.vmware.vapi.std.errors.unauthenticated
    --> Messages:
    -->    vapi.security.authentication.invalid<Unable to authenticate user>
    ..
    ..
    .. YYYY-MM-DDTHH:MM:SS.###-##:## info vpxd[10034] [Originator@6876 sub=Default opID=opId-18b14-105289-d9] [VpxLRO] -- ERROR task-6215 -- host-9421 -- vim.HostSystem.enterMaintenanceMode: vim.fault.InvalidState:
    --> Result:
    --> (vim.fault.InvalidState) {
    -->    faultCause = (vmodl.MethodFault) null, 
    -->    faultMessage = (vmodl.LocalizableMessage) [
    -->       (vmodl.LocalizableMessage) {
    -->          key = "com.vmware.cdrs.maintenancemode.wcp.entermaintenancemode", 
    -->          arg = <unset>, 
    -->          message = <unset>

  • Additionally, Software-Defined Data Center (SDDC) Manager workflows fail with the error:

FAILED_TO_GET_WCP_CLUSTER_STATUS
Failed to get Workload Management cluster status for vCenter <VC_FQDN>

 

Environment

VMware vCenter Server 7.0.x
VMware vCenter Server 8.0.x
VMware Cloud Foundation 4.x

Cause

This issue occurs when the Workload Control Plane (WCP) solution user certificate is invalid. This happens due to a duplicate solution user certificate for the WCP service in a vCenter linked mode setup after utilizing the certificate-manager utility. It also occurs if the WCP solution user certificate expires, as early versions defaulted to a two-year validity for the WCP service compared to the standard ten-year validity for other solution users.

Resolution

Note: Take an offline (powered-off) snapshot of the vCenter Server before proceeding. If operating in Enhanced Linked Mode (ELM), take offline snapshots of all replicating nodes.


Method  1: Replace using vCert

  1. Download the script from KB 385107 and upload it to the /root or /tmp directory on the vCenter Server.

  2. Run the script using the command:
    ./vCert.py.

  3. Provide the Single Sign-On (SSO) administrator credentials (e.g., [email protected]).

  4. Select Option 3 from the main menu to manage certificates.

  5. Select Option 2 to manage Solution User certificates.

  6. Select Option 1 to replace with VMCA signed certificates. Alternatively, select the specific option for custom Certificate Authority (CA)-signed certificates if applicable to your environment.

 

Method  2: Replace Manually

  1. SSH into the vCenter Server where you need to repair the WCP service.

  2. Get the unique Machine ID and hostname by running:
    /usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost hostname -f

  3. Generate the WCP solution user key by running:
    /usr/lib/vmware-vmca/bin/certool --server localhost --genkey --privkey=/tmp/wcp.key --pubkey=/tmp/wcp.pub

  4. Generate the WCP solution user certificate by running:
    /usr/lib/vmware-vmca/bin/certool --server=localhost --genCIScert --privkey=/tmp/wcp.key --cert=/tmp/wcp.crt --Name=wcp --Hostname=<VC_FQDN>

  5. Get the WCP service name using the directory command-line interface.The default name is wcp-<machine id>:
    /usr/lib/vmware-vmafd/bin/dir-cli service list

  6. Update the WCP service with the newly generated WCP certificate:
    /usr/lib/vmware-vmafd/bin/dir-cli service update --name <insert wcp service name from the service list> --cert /tmp/wcp.crt

  7. Delete the stale WCP solution user entry from the VMware Endpoint Certificate Store (VECS) by running:
    /usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store wcp --alias wcp -y /usr/lib/vmware-vmafd/bin/vecs-cli force-refresh

  8. Update the new WCP solution user certificate to the VECS store by running:
    /usr/lib/vmware-vmafd/bin/vecs-cli entry create --store wcp --alias wcp --cert /tmp/wcp.crt --key /tmp/wcp.key

  9. Verify that the WCP certificate updates successfully. The Subject must contain the unique CN as updated in wcp.cfg, along with a new Issue and Expiration date:
    /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store wcp --alias wcp --text

  10. Restart all services on the vCenter Server to apply the changes:
    service-control --stop --all && service-control --start --all

Powered by Wolken Software