Unable to login to the TKC guest clusters via jump host using an active directory user

Products

• VMware vSphere Kubernetes Service
• VMware vCenter Server

Issue/Introduction

• After a power outage, unable to login to the TKC guest clusters via jump host using a developer user ID and it's a AD user

  • ~ # kubectl-vsphere login --server <IP_Address> --insecure-skip-tls-verify --tanzu-kubernetes-cluster-namespace intl -u <[email protected]> --tanzu-kubernetes-cluster-name <cluster_name>
    
    Password:
    ERRO[0004] invalid or missing credentials

• Unable to login to VC as well using the same user, while the user is added to the vCenter global permissions  
• vCenter is configured with AD over Integrated Windows Authentication (IWA)
• Domain Account (AD login) fails with "Invalid Credentials" error message in vSphere Client
• /var/log/vmware/sso/websso.log in vCenter shows below similar errors 

  • YYYY-MM-DDTHH:MM:SS.Z WARN  websso[82:tomcat-http--44] [CorId=d######-####-####-####-########4] [com.vmware.identity.idm.server.ServerUtils] cannot bind connection: [ldap://######@domainname, CN=Username\\,user.id,CN=Users,DC=corp,DC=domain,DC=com]
    YYYY-MM-DDTHH:MM:SS.Z INFO  websso[82:tomcat-http--44] [CorId=d######-####-####-####-########4] [com.vmware.identity.diagnostics.VmEventAppender] EventLog: source=[VMware Identity Server], tenant=[vsphere.local], eventid=[USER NAME PWD AUTH FAILED], category=[VMEVENT_CATEGORY_STS], text=[Failed to authenticate principal [username @domain.com]. Login failed], detailText=[Login failed], correlationId=[d######-####-####-####-########4], timestamp=[1757400051669]
    YYYY-MM-DDTHH:MM:SS.Z INFO  websso[82:tomcat-http--44] [CorId=d######-####-####-####-########4] [com.vmware.identity.idm.server.IdentityManager] Authentication failed for user [username @domain.com] in tenant [vsphere.local] in [411] milliseconds with provider [domain.com] of type [com.vmware.identity.idm.server.provider.ldap.LdapWithAdMappingsProvider]
    YYYY-MM-DDTHH:MM:SS.Z ERROR websso[82:tomcat-http--44] [CorId=d######-####-####-####-########4] [com.vmware.identity.samlservice.impl.SAMLAuthnResponseSender] Responded with ERROR 401 message Invalid credentials
    YYYY-MM-DDTHH:MM:SS.Z INFO  websso[82:tomcat-http--44] [CorId=d######-####-####-####-########4] [com.vmware.identity.BaseSsoController] End processing. Session not created.

Environment

vSphere with Tanzu

VMware vCenter Server

Resolution

vCenter login with AD users fails because the secure channel (trust) between vCenter’s machine account and Active Directory is broken, most often due to password sync issues, time skew, or DNS failures.

To resolve this issue, take a snapshot of the VCSA VM and the disjoin and re-join VC to the active directory domain by following below

• • Take snapshot of VC 
  • Run the below command to extract the existing domain details
    • /opt/likewise/bin/domainjoin-cli query
  • Run the below command to disjoin the vCenter Server Appliance (VCSA) from the windows domain
    • /opt/likewise/bin/domainjoin-cli leave
  • Reboot the vCenter appliance
  • Run below command in order to join the appliance to the windows domain
    • /opt/likewise/bin/domainjoin-cli join <Active_Directory_Domain> <Domain_Administrator> <Password>
    • the above join step can also be done through vSphere UI 
  • Reboot the vCenter appliance
  • Run the below command to verify the domain details
    • /opt/likewise/bin/domainjoin-cli query

You should successfully be able to login with the AD accounts to both vCenter and the k8s guest clusters via jumphost.

Additional Information

Refer to KB article Unable to Login to vCenter Server with AD domain accounts as it fails with "Invalid Credentials" Error