Apply Changes fails due to error NoCredentialProviders: no valid providers in chain.

Products

• VMware Tanzu Application Service

Issue/Introduction

Apply Changes process fails with the following error message:

Task xxx | hh:mm:ss | Error: Action Failed get_task: Task xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx result: Compiling package golang-1-linux: Fetching package golang-1-linux: Fetching package blob xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx: Getting blob from inner blobstore: Getting blob from inner blobstore: Shelling out to bosh-blobstore-s3 cli: Running command: 'bosh-blobstore-s3 -c /var/vcap/bosh/etc/blobstore-s3.json get xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx /var/vcap/data/tmp/bosh-blobstore-externalBlobstore-GetXXXXXXXXXX', stdout: '', stderr: 'YYYY/MM/DD hh:mm:ss performing operation get: NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors

NoCredentialProviders: no valid providers in chain. Deprecated.

Environment

Blobstore on AWS.

Cause

This is can be a permissions related issue on the key/secret used to access the blobstore, issues after enabling IAM services for login or S3 bucket access permissions.

The following is an example of what would be visible in the change logs if the S3 blobstore authentication method is changed from access key and secret to enabling IAM:

    - name: blobstore
      properties:
        select_directories_to_backup:
          buildpacks:
-           aws_access_key_id: XXXXXXXXXXXXXXXXXXX
-           aws_secret_access_key: "((/opsmgr/cf-xxxxxxxxxxxxxxxxxxxx/system_blobstore/external/secret_key.value))"
+           use_iam_profile: true
          droplets:
-           aws_access_key_id: XXXXXXXXXXXXXXXXXXX
-           aws_secret_access_key: "((/opsmgr/cf-xxxxxxxxxxxxxxxxxxxx/system_blobstore/external/secret_key.value))"
+           use_iam_profile: true
          packages:
-           aws_access_key_id: XXXXXXXXXXXXXXXXXXX
-           aws_secret_access_key: "((/opsmgr/cf-xxxxxxxxxxxxxxxxxxxx/system_blobstore/external/secret_key.value))"
+           use_iam_profile: true

Resolution

Review the AWS bucket permissions. If the issue is happening after enabling IAM review the permissions or consider reverting to using AWS access keys.