kubectl vsphere login fails with "Failed to get available workloads: bad gateway"

Products

• VMware vSphere ESXi
• VMware vSphere with Tanzu

Issue/Introduction

Symptoms:

When trying to connect to Supervisor clusters as Active Directory User, kubectl login fails with

kubectl vsphere login --server <KUBERNETES-CONTROL-PLANE-IP-ADDRESS> -u <AD USER>
level=fatal msg="Failed to get available workloads: bad gateway\nPlease contact your vSphere server administrator for assistance." Error: exit status 1

 

Environment

VMware vSphere 7.0 with Tanzu

Cause

The Auth proxy logs on Supervisor VM shows the login failing with 502 error for the AD user:

2022-08-09T12:55:41.893672642Z stderr F ERROR:vclib.wcp:[140194735557136] WCP request failed.
:
2022-08-09T12:55:41.894080177Z stderr F INFO:server:[140194735557136] "127.0.0.1" - - [09/Aug/2022:12:55:41 +0000] "GET /wcp/workloads HTTP/1.0" 502 46 "-" "kube-plugin-vsphere bld 18647806 - cln 9232193" "<AD USER>"

 

The vpxd-svcs.log logs show that Principal Name configured in AD server is invalid. The User is a member of an Active Directory group whose name contains a special character, ie @ in this case.

2022-08-09T12:55:41.889Z [tomcat-exec-106 ERROR com.vmware.cis.core.authz.accesscontrol.impl.AuthzServiceBaseImpl opId=ca4aaf01-8bb7-4f9e-b4b8-0e3679659899] Store Exception
java.lang.IllegalArgumentException: Invalid principal name <USER GROUP>@NAME@DOMAIN>
  at com.google.common.base.Preconditions.checkArgument(Preconditions.java:217)
  at com.vmware.cis.core.util.NormalizedPrincipalHelper.getNormalizedPrincipal(NormalizedPrincipalHelper.java:30)
:

 

Resolution

This issue has been resolved in vCentre Server 7.0 Update 3f

Workaround:

As a workaround, please do not use Active Directory groups with a "@" character in the name. If this is not an option, remove affected users from such groups.