Apply Changes failing in Operations Manager (Ops Man) with the error “Putting Tile Credentials into Credhub”

Products

• Operations Manager

Issue/Introduction

The UAA token used to authenticate with CredHub has a 10 minute expiry. We have a retry logic in place which gives an additional 10 minutes for this operation - “Copying credentials to CredHub during Apply Changes.”

If putting the Tile Credentials into CredHub takes more than 20 (10+10) minutes, applying changes in Ops Man fails.

Note: The Tanzu Ops Manager (Ops Man) team looking into a few options of making this more efficient and fail-safe. 

Resolution

Until a permanent fix is released, the following workaround can be implemented. Please note that you need the decryption password if you proceed with the steps below.

1. Change user to tempest-web with the following command: sudo su - tempest-web

2. Increase the number of retries allowed before the step fails. This can be done by editing /home/tempest-web/tempest/web/lib/product_credentials_credhub_repository.rb on the Ops Manager VM and changing the following line:

retries_remaining = 1 to retries_remaining = 3

3. Become root with the following command: sudo -i

4. Restart tempest-web with the following command: service tempest-web restart, and wait till it starts.

5. Next, you will have to enter the decryption passphrase.

6. Apply Changes using Ops Man.
 

Explanation

When you change the retry count from 1 to 3, it provides a total of 40 minutes ( [10 + (3*10)] ) versus the original 20 minutes for this operation of copying the Tile credentials into CredHub to complete.

Note: The Tanzu Ops Manager (Ops Man) team looking into a few options of making this more efficient and fail-safe. There will be a new Ops Man release with better mitigations. This article will be updated to reflect those details.