New supervisor Activation fails at Ready to complete stage with error "A general system error occurred. Error message: 400 Bad Request: {"error_type":"INVALID_ARGUMENT","messages":[{"args":["messages","

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Configuring vSphere Supervisor with the "vSphere Networking with VPC" Networking Stack option fails at the "Ready to complete" stage with the following error.
  
  A general system error occurred. Error message: 400 Bad Request: {"error_type":"INVALID_ARGUMENT","messages":[{"args":["messages","com.vmware.vapi.std.errors.unauthenticated"],"default_message":"Could not convert field 'messages' of structure 'com.vmware.vapi.std.errors.unauthenticated'","id":"vapi.bindings.typeconverter.tovalue.struct.field.error"},{"args":["args","com.vmware.vapi.std.localizable_message"],"default_message":"Required field 'args' of structure 'com.vmware.vapi.std.localizable_message' is missing","id":"vapi.bindings.typeconverter.tovalue.struct.field.missing"}]}.
  
  
• Per /var/log/vmware/wcp/wcpsvc.log, it fails to list the Zone-Cluster association.
  
  error wcp [zones/zones.go] [opID=<ID>] Failed list Zone-Cluster Association for zone domain-c10 and cluster domain-c<ID>: 400 Bad Request: {"error_type":"INVALID_ARGUMENT","messages":[{"args":["messages","com.vmware.vapi.std.errors.unauthenticated"],"default_message":"Could not convert field 'messages' of structure 'com.vmware.vapi.std.errors.unauthenticated'","id":"vapi.bindings.typeconverter.tovalue.struct.field.error"},{"args":"args","com.vmware.vapi.std.localizable_message"],"default_message":"Required field 'args' of structure 'com.vmware.vapi.std.localizable_message' is missing","id":"vapi.bindings.typeconverter.tovalue.struct.field.missing"}]}
error wcp [wcp/supervisor.go] [opID=<ID>] Failed to create deployment target for cluster: domain-c10, err: 400 Bad Request: {"error_type":"INVALID_ARGUMENT","messages":[{"args":["messages","com.vmware.vapi.std.errors.unauthenticated"],"default_message":"Could not convert field 'messages' of structure 'com.vmware.vapi.std.errors.unauthenticated'","id":"vapi.bindings.typeconverter.tovalue.struct.field.error"},{"args":["args","com.vmware.vapi.std.localizable_message"],"default_message":"Required field 'args' of structure 'com.vmware.vapi.std.localizable_message' is missing","id":"vapi.bindings.typeconverter.tovalue.struct.field.missing"}]}


• On checking the status of the services on the vCenter Server Appliance, the vmware-trustmanagement service is observed to be in a stopped state. An example output is included below.
  
  Running:
applmgmt lookupsvc lwsmd observability observability-vapi pschealth
vc-ws1a-broker vlcm vmafdd vmcad vmdird vmware-analytics vmware-certificateauthority vmware-certificatemanagement vmware-cis-license
vmware-content-library vmware-eam vmware-envoy vmware-envoy-hgw vmware-envoy-sidecar vmware-envoy-system-proxy vmware-hvc vmware-infraprofile
vmware-perfcharts vmware-postgres-archiver vmware-rhttpproxy vmware-sca vmware-sps vmware-stsd vmware-topologysvc vmware-updatemgr vmware-vapi-endpoint
vmware-vdtc vmware-vmon vmware-vpostgres vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-ui vstats vtsdb wcp

  Stopped:
 vmcam vmware-imagebuilder vmware-netdumper vmware-rbd-watchdog vmware-trustmanagement vmware-vcha

• In /var/log/vmware/trustmanagement/tms-prestart.log, the service account creation fails.
  
  INFO:root:Setting-up service-account for service: lookupsvc
INFO:root:Service account for lookupsvc is invalid, Will go ahead with service account recreation.
ERROR:__main__:Failed to set up service account for trustmanagement


Environment

vSphere Kubernetes Service
VMware Cloud Foundation 9.0.x
VMware vCenter Server 9.0.x

Cause

During the first step of compatibility check for NSX-VPC, WCP service checks to see where vCenter Server is hosted. It relies on the identity service (vmware-trustmanagement) to complete the same. Because the vmware-trustmanagement service is in a stopped state, it causes the deployment targets for zones not getting created due to the failure to list zone/cluster associations with a 400 bad request.

Resolution

This is a known issue in vCenter Server 9.0 where the re-creation of service account by trustmanagement service can fail in some special scenarios. The same is being worked upon internally by Broadcom and shall be addressed in an upcoming release.

Meanwhile, the workaround is to restart the vmware-trustmanagement service manually. Run the following command in the vCenter appliance SSH

service-control --restart vmware-trustmanagement