kubectl vsphere login fails with "Failed to get available workloads: bad gateway"

Products

• VMware vSphere ESXi
• VMware vSphere Kubernetes Service

Issue/Introduction

• Attempting to connect to Supervisor clusters as Active Directory User, kubectl login fails with the following error:
  
  kubectl vsphere login --server <KUBERNETES-CONTROL-PLANE-IP-ADDRESS> -u <AD USER> level=fatal msg="Failed to get available workloads: bad gateway\nPlease contact your vSphere server administrator for assistance." Error: exit status 1

• The Auth proxy logs on Supervisor VM shows the login failure with 502 error for the AD user:
  
  YYYY-MM-DDTHH:MM:SSZ stderr F ERROR:vclib.wcp:[140194735557136] WCP request failed. YYYY-MM-DDTHH:MM:SSZ stderr F INFO:server:[140194735557136] "127.0.0.1" - - "GET /wcp/workloads HTTP/1.0" 502 46 "-" "kube-plugin-vsphere bld 18647806 - cln 9232193" "<AD USER>"

• The vpxd-svcs.log logs show that Principal Name configured in AD server is invalid. The User is a member of an Active Directory group whose name contains a special character, ie @ in this case.
  
  YYYY-MM-DDTHH:MM:SSZ [tomcat-exec-106 ERROR com.vmware.cis.core.authz.accesscontrol.impl.AuthzServiceBaseImpl opId=<op_Id>] Store Exception java.lang.IllegalArgumentException: Invalid principal name <USER GROUP>@NAME@DOMAIN>   at com.google.common.base.Preconditions.checkArgument(Preconditions.java:217)   at com.vmware.cis.core.util.NormalizedPrincipalHelper.getNormalizedPrincipal(NormalizedPrincipalHelper.java:30)

Environment

vSphere Kubernetes Service

Resolution

This issue has been resolved in vCenter Server 7.0u3f

Workaround:

Do not use Active Directory groups with a "@" character in the name. If this is not an option, remove affected users from such groups.