AWS IAM Key Rotation Procedure for VMware Tanzu Application Service (TAS) for VMs

Products

• Operations Manager

Issue/Introduction

Changing access keys, which consist of an access key ID and a secret access key, on a regular schedule is a well-known security best practice because it shortens the period an access key is active and therefore reduces the business impact if they are compromised.

Having an established process that is run regularly also ensures the operational steps around key rotation are verified, so changing a key is never a scary step.

Environment

Product Version: 2.7

Resolution

When rotating AWS access keys in TAS for VMs, there are 3 locations that need to be updated. Locations #1 and #3 are editable and can be easily changed within Ops Manager.

1. BOSH Director Tile ---> AWS Config
2. BOSH Director Tile ---> Director Config ---> S3 Compatible Blobstore (Write-Protected)
3. TAS Tile ---> File Storage ---> Configure your Cloud Controller's filesystem

Location #2 (S3 Blobstore) is write-protected in Ops Manager after the first deploy. In order to update the access keys for this field, we need to modify the installation.yml by following the steps outlined below:

1. SSH into the Ops Manager VM. For information on how to do this, refer to the following documentation: https://docs.pivotal.io/platform/2-7/customizing/trouble-advanced.html#ssh

2. On the command line, navigate to the scripts directory:​​​​​​

cd /home/tempest-web/tempest/web/scripts/

3. Run the below command to decrypt the installation.yml and make a temp copy of the file. When prompted for a passphrase, enter the decryption passphrase you created when you launched Ops Manger for the first time.

sudo -u tempest-web SECRET_KEY_BASE="s" ./decrypt /var/tempest/workspaces/default/installation.yml /tmp/installation.yml

4. Open /tmp/installation.yml and update the access keys.

5. If you plan to make changes, make a backup of the original installation YAML file:

cp /var/tempest/workspaces/default/installation.yml ~/installation-orig.yml

6. If you have made changes to your copy of the installation YAML file, you must encrypt it and overwrite the original with it. When prompted, enter the passphrase:

sudo -u tempest-web SECRET_KEY_BASE="s" RAILS_ENV=production /home/tempest-web/tempest/web/scripts/encrypt /tmp/installation.yml /var/tempest/workspaces/default/installation.yml

7. Delete the temp copy of the decrypted file:

rm /tmp/installation.yml

8. Restart Ops Manager web interface

sudo service tempest-web stop && sudo service tempest-web start

To complete the key rotation:

1. Navigate back to Ops Manager in a browser and enter your decryption passphrase. Log in to Ops Manager
2. Navigate to the Director Config and enable 'Recreate All VMs'. This will ensure the new properties are pushed to each VM
3. Click Review Changes, then Apply Changes.
4. If Ops Manager cannot load your changes, see the Revert To Your Backup section to restore your previous settings in installation.yml.