Post Start Failure During Tanzu Application Service Tile Installation with AWS using IAM instance profile

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

When installing a new TAS deployment into AWS and using IAM instance profile (under file storage), TAS tile installation might fail with the following error messages:

Task xxx | TIME | L executing post-start: cloud_controller/ID (0) (canary) (00:02:31)
                   L Error: Action Failed get_task: Task ID result: 1 of 2 post-start scripts failed. Failed Jobs: cloud_controller_ng. Successful Jobs: bosh-dns.
Task xxx | TIME | Error: Action Failed get_task: Task ID result: 1 of 2 post-start scripts failed. Failed Jobs: cloud_controller_ng. Successful Jobs: bosh-dns.

Looking in the logs cloud_controller_ng job has the following errors:

Missing required arguments: aws_access_key_id, aws_secret_access_key

[fog][WARNING] Unable to fetch credentials: Expected(200) <=> Actual(404 Not Found)

{"timestamp":"TIME","message":"Encountered error: Missing required arguments: aws_access_key_id, aws_secret_access_key\n/var/vcap/data/packages/cloud_controller_ng/ID/cloud_controller_ng/lib/cloud_controller/blobstore/fog/error_handling_client.rb:68:in `rescue in ...

In order to use AWS with IAM instance profile there are certain steps that need to be completed in order to apply such configuration because even if aws_secret_access_key is selected, the switch will override the value as specified in our documentation.

Environment

Product Version: 2.1

Resolution

Follow the doc linked below:

https://docs.cloudfoundry.org/deploying/common/cc-blobstore-config.html#fog-aws-iam

1. Configure an additional cloud-controller IAM role in AWS.

2. In your BOSH cloud config, create a VM extension to add the IAM Instance Profile you created to VMs using the extension. To complete this part, om can be utilized.

Install om; it can be it installed on your mac, linux jumpbox, or ops manager. Create a var file cat vars.yaml:

target: https://<opsmgr-FQDN>
skip-ssl-validation: true
username: admin
password: <OPSMANPASS>
connect-timeout: 5
request-timeout: 1800
trace: false
decryption-passphrase: ""

Confirm om is working:

om --env vars.yaml products --staged

Add vm extension where "cloud-controller" is the name of the IAM role created earlier:

om  --env vars.yaml create-vm-extension --name 'cloud-controller-iam' --cloud-properties '{ "iam_instance_profile": "cloud-controller" }'

Confirm the creation is successful:

om --env vars.yaml curl /api/v0/staged/vm_extensions

Output:

{
  "vm_extensions": [
    {
      "name": "cloud-controller-iam",
      "cloud_properties": {
        "iam_instance_profile": "cloud-controller"
      }
    }
  ]
}

Export TAS config to a file:

om --env vars.yaml staged-config -p cf > config.yaml

Make a copy of the file for backup and restore. Then edit the file where there are two sections to be modified cloud_controller and cloud_controller_worker under the additional_vm_extensions add the VM extension name created earlier:

cat config.yaml | grep cloud-controller-iam -B3
  cloud_controller:
    max_in_flight: 1
    additional_networks: []
    additional_vm_extensions: [cloud-controller-iam]
--
  cloud_controller_worker:
    max_in_flight: 1
    additional_networks: []
    additional_vm_extensions: [cloud-controller-iam]

Update the config with the modified settings:

om --env vars.yaml configure-product -p cf -c config.yaml

Note: The TAS tile has to be fully configured from the UI or pipeline before applying this update.

Initiate apply change from UI or om:

om --env vars.yaml apply-changes -p cf 

To confirm the successful deployment, you should see the instances of cloud controller visible under the policy in AWS.