Requests to UAA failing with "UAA intentionally in limited mode, operation not permitted. Please try later."

Products

• VMware Tanzu Application Service for VMs
• VMware Tanzu Kubernetes Grid

Issue/Introduction

When attempting to perform a request to the TAS UAA, it returns the following:
 

UAA intentionally in limited mode, operation not permitted. Please try later.

This error can be returned by a variety of components that interact with UAA, such a service tiles (MySQL, SCS, etc)

Environment

Product Version: 2.13

Resolution

UAA is placed into limited mode when BBR (Bosh Backup and Restore) is running against the TAS deployment. It does this by placing a file in the following directory: /var/vcap/data/uaa/bbr_limited_mode.lock

https://github.com/cloudfoundry/uaa-release/blob/98d5d84976b25e5eb667d4be18e33c3bad677ae6/jobs/uaa/spec#L216-L221

Once BBR has completed, this file is removed, and UAA is returned to regular functionality.

However, in the event the previous attempt for BBR fails, this cleanup may not occur, and the file is not removed. In this case, you will need to clean it up manually by following the steps outlined here:

https://docs.vmware.com/en/VMware-Tanzu-Operations-Manager/3.0/vmware-tanzu-ops-manager/install-backup-restore-backup-pcf-bbr.html#canceling-a-backup-27

This will remove the file lock file from the UAA VMs.