Workload Management shows "We encountered an error and are unable to reach the service" in vSphere
Article ID: 323423
Updated On:
Products
Issue/Introduction
1. One of the below error messages are seen in Workload Management in vSphere:
"None of the hosts connected to this vCenter are licensed for Workload Management"
"We encountered an error and are unable to reach the service"
2. Tanzu Kubernetes Clusters are not visible in vSphere inventory.
3. The below entries are present in /var/log/vmware/wcpsvc.log:
2023-01-19T01:34:40.701Z debug wcp [ssolib/sts.go:87] [opID=63c89b11] Getting HOK signer; store: wcp, alias: wcp
2023-01-19T01:34:40.75Z error wcp [ssolib/sts.go:122] [opID=63c89b11] STS Issue HOK request failed; err: ns0:FailedAuthentication: Invalid credentials
2023-01-19T01:34:40.75Z error wcp [kubelib/auth.go:115] [opID=63c89b11] Failed to obtain JWT for wcp: ns0:FailedAuthentication: Invalid credentials
2023-01-19T01:34:40.75Z error wcp [kubelib/auth.go:160] [opID=63c89b11] Failed to obtain JWT: ns0:FailedAuthentication: Invalid credentials
2023-01-19T01:34:40.751Z debug wcp [kubelib/auth.go:203] [opID=63c89b11] Auth roundtripper: retry request. Response statusCode: 401
4. The below entries are present in /var/log/vmware/sso/vmware-identity-sts.log:
2023-01-19T16:11:52.828Z INFO sts[77:tomcat-http--39] [CorId=49e3e12e-6697-4ac9-a848-62186be6f693] [com.vmware.identity.sts.ws.SOAPFaultHandler] Returning a SOAP Fault with code: ns0:FailedAuthentication and description: Invalid credentials
2023-01-19T16:11:52.736Z INFO sts[84:tomcat-http--46] [CorId=62b603b1-f3e2-4332-82d8-8c6aded22dfe] [com.vmware.identity.sts.ws.SOAPFaultHandler] Returning a SOAP Fault with code: ns0:FailedAuthentication and description: Invalid credentials
2023-01-19T16:11:52.826Z INFO sts[77:tomcat-http--39] [CorId=49e3e12e-6697-4ac9-a848-62186be6f693] [com.vmware.identity.sts.InvalidCredentialsException] Censored exception
com.vmware.identity.sts.InvalidCredentialsException: Solution user cert is not valid.
Environment
Cause
The WCP Solution User certificate is expired.
Resolution
1. Login to an SSH session to vCenter, and run the below command to check if the WCP Solution User certificate is expired:
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store wcp --text | grep "Not After"
2. If the certificate is expired, replace the certificate by performing the following steps.
A. Gather the unique machine id, and the hostname by running the below commands on vCenter
# /usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost
# hostname
B. Update the fields in red in the WCP Solution User configuration file, with desired values found with commands in previous step
# vi /var/tmp/vmware/wcp.cfg
Country =
Name = wcp-<insert unique machine id>
Organization =
OrgUnit = mID-<insert unique machine id>
State =
Locality =
IPAddress =
Email =
Hostname = <FQDN fetched from output of hostname command>
C. Generate WCP Solution User Key
# /usr/lib/vmware-vmca/bin/certool --server localhost --genkey --privkey=/root/wcp.key --pubkey=/root/wcp.pub
D. Generate the WCP Solution User Certificate
# /usr/lib/vmware-vmca/bin/certool --server=localhost --gencert --privkey=/root/wcp.key --config=/var/tmp/vmware/wcp.cfg --cert=/root/wcp.crt
E. Get WCP service name from dir-cli ; default name - wcp-<machine id>
# /usr/lib/vmware-vmafd/bin/dir-cli service list
F. Update WCP service with new WCP certificate
/usr/lib/vmware-vmafd/bin/dir-cli service update --name <insert wcp service name from the service list> --cert /root/wcp.crt
G. Delete the WCP Solution User entry from the vecs store
# /usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store wcp --alias wcp -y
# /usr/lib/vmware-vmafd/bin/vecs-cli force-refresh
H. Update the WCP Solution User certificate to vecs store
# /usr/lib/vmware-vmafd/bin/vecs-cli entry create --store wcp --alias wcp --cert /root/wcp.crt --key /root/wcp.key
I. Verify the WCP certificate is updated; Subject should contain unique CN as updated in wcp.cfg
# /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store wcp --alias wcp --text
J. Restart all vCenter services
# service-control --stop --all && service-control --start --all && service-control --status
3. Verify that the certificate no longer shows expired:
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store wcp --text | grep "Not After"
Feedback
