FailedLoginAttempts SupervisorControlPlaneVM Login Issues for wcp-vmimageserviceop-user

Products

• VMware vSphere Kubernetes Service
• VMware vCenter Server 8.0

Issue/Introduction

In a vSphere with Tanzu environment, the following symptoms are observed:

• Syslog or Log Insight Alerts:** Frequent FailedLoginAttempts for the user wcp-vmimageserviceop-user-<UUID>@vsphere.local.
• vCenter Events:** Log events indicate vim.event.BadUsernameSessionEvent (unknown or invalid username) originating from a SupervisorControlPlaneVM source.
• Service Availability:** Intermittent 503 Service Unavailable or 502 Bad Gateway errors when accessing `/sdk` or performing kubectl vsphere login operations.
• Operator Status:** The imageregistry-operator may show reconciliation failures or remain in a Configuring state.

Environment

• VMware vCenter Server 8.0.x
• vSphere Supervisor

Cause

This issue is caused by the imageregistry-operator caching stale vCenter credentials. When the wcp-vmimageserviceop-user service account password rotates or the connection to the vCenter server service (vpxd) is briefly interrupted (returning 503 errors), the operator may attempt to authenticate using stale credentials. 

Because the operator does not implement a back-off delay between failed attempts, it can trigger multiple invalid logins in rapid succession, exceeding the vCenter Single Sign-On (SSO) lockout threshold and locking the service account.

Resolution

A permanent fix for this behavior will be part of the vCenter Server 9.1 release. 

To work around the issue, please apply the steps below to unlock the account and refresh the operator.

1. Verify and unlock the service account:
   • Open an SSH connection to the vCenter Server Appliance (VCSA) and login with the root account.
   • Run the following command to find the full name of the affected service account:

     # cat /var/log/vmware/vmdird/vmdird.log | grep vmimageservice

   • Check the lockout status (replace <ID> against the ID received in the step above):

     # /usr/lib/vmware-vmafd/bin/dir-cli user find-by-name --account wcp-vmimageserviceop-user-<ID> --level 2

   • If the command returns an Account locked: TRUE, unlock the account (again replace <ID> as before):

     # /opt/likewise/bin/ldapmodify -x -D cn=Administrator,cn=Users,dc=vsphere,dc=local -W <<EOF
     dn: cn=wcp-vmimageserviceop-user-<ID>,cn=serviceprincipals,dc=vsphere,dc=local
     changetype: modify
     replace: userAccountControl
     userAccountControl: 0
     EOF

2. Restart the Image Registry Operator:
   • Connect to the Supervisor Cluster context using kubectl.
   • Scale the operator deployment down to 0 and back to its original replica count (usually 2) to clear the credential cache:

     # kubectl scale deploy -n vmware-system-imageregistry vmware-system-imageregistry-controller-manager --replicas=0
     
     # kubectl scale deploy -n vmware-system-imageregistry vmware-system-imageregistry-controller-manager --replicas=2

Additional Information

See also Workload Cluster Upgrade Stuck on New Control Plane Machine Provisioning due to Locked ImageRegistryOperator User