App Autoscaler Errand keeps failing because the UAA admin account gets locked due to multiple failed login attempts

Products

• VMware Tanzu Platform - Cloud Foundry

Issue/Introduction

The TAS UAA Admin password has been rotated accordingly similar to the procedure in the Rotating TAS UAA admin user & client password KB, but during the Apply Changes, the App Autoscaler Errand failed.  The failure occurs because the UAA admin account was locked due to multiple failed login attempts.

Environment

VMware Tanzu Platform - Cloud Foundry

Tanzu Observability by Wavefront Nozzle

Cause

The cc_exporter job (from the Tanzu Observability by Wavefront Nozzle deployment) was still using the old admin password, because the wavefront-nozzle tile has not been updated yet.  The Apply Changes updates the TAS deployment first before the Wavefront Nozzle deployment.

Resolution

As a workaround to recover, the following steps can be executed:

1. Using the bosh CLI, SSH to the tas-exporters VM and stop the cc_exporter job manually using monit. This should stop the job from repeatedly logging in with the old invalidated password and prevent the account from being locked again.
   

   monit stop cc_exporter

2. Unlock the admin account using the UAAC CLI.  You can use the Ops Manager VM to perform this step, as it should have the CLI installed.  
   

   uaac target https://uaa.<SYS_DOMAIN> --skip-ssl-validation
   uaac token client get admin -s "<UAA_ADMIN_CLIENT_PASSWORD>"
   uaac user unlock admin

   Note that "SYS_DOMAIN" is the Tanzu Platform for Cloud Foundry System Domain, and "UAA_ADMIN_CLIENT_PASSWORD" is the password from Tanzu Platform for Cloud Foundry Credentials tab->UAA->Admin Client Credentials.

3. Run Apply Changes on all the tiles including the Tanzu Observability by Wavefront Nozzle.  This should update the tas-exporters' cc_exporter configuration with the new password.  The App Autoscaler errand should also complete successfully this time because the admin is not locked anymore.

 

Additional Information

The Rotating TAS UAA admin user & client password KB has been updated to include a step, if the Tanzu Observability by Wavefront Nozzle is installed, to prevent this issue from happening.