Stemcell Update Causes CPI Error after Enabling AWS EBS Encryption

Products

• Operations Manager

Issue/Introduction

Amazon EBS encryption uses AWS Key Management Service (KMS) and customer master keys (CMK) when creating encrypted volumes and snapshots.

Encryption operations occur on the servers that host EC2 instances, ensuring the security of both data-at-rest and data-in-transit between an instance and its attached EBS storage.

Customers can enable EBS encryption for TAS deployments by following the below documentation. However, there are additional permission requirements that you need to add.

https://docs.pivotal.io/platform/2-10/customizing/cloudform-om-ebs-config.htm
A stemcell update causes a CPI Error after enabling AWS EBS encryption.

Symptoms:
After enabling EBS Encryption for TAS deployments on AWS, you will need to update each VM with a new stemcell update. During the process, the update may fail due to a vague CPI error:

Updating deployment:
 Expected task '1141936' to succeed but state is 'error'
Exit code 1
Task 1141936 | 20:13:53 | Compiling packages: golang-1-linux/61eb5fdbd273e932841c05fa7897ca0075a54b64eb0023b4eda396c5077bf3a4 (00:00:27)
                    L Error: Unknown CPI error 'Unknown' with message 'stopped waiting, encountered a failure state' in 'create_vm' CPI method (CPI request ID: 'cpi-223975')

If you take a look at the debug output for that particular task, you will find additional information from AWS regarding 'FailureStateError'.

https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/Waiters/Errors/FailureStateError.html

D, [2020-08-31T20:13:33.417191 #5783] DEBUG -- [req_id cpi-486912]: [Aws::EC2::Client 400 0.049666 0 retries] describe_instances(instance_ids:["i-0261468c53393308f"]) Aws::EC2::Errors::InvalidInstanceIDNotFound The instance ID 'i-0261468c53393308f' does not exist

W, [2020-08-31T20:13:53.729197 #5783]  WARN -- [req_id cpi-486912]: Failed to configure instance 'i-0261468c53393308f': #<Aws::Waiters::Errors::FailureStateError: stopped waiting, encountered a failure state>

Environment

Product Version: 2.9

Cause

The CPI is unable to update the instance because the IAM policy being used doesn't have the correct access and policies attached to it.

Resolution

In order to perform specific tasks to encrypted EBS volumes, you need to add the below AWS policies:

{
    "Sid": "RequiredIfUsingHeavyStemcells",
    "Effect": "Allow",
    "Action": [
        "ec2:RegisterImage",
        "ec2:DeregisterImage"
    ],
    "Resource": "*"
},
{
    "Sid": "RequiredIfEncryptingStemcells",
    "Effect": "Allow",
    "Action": [
        "ec2:CopyImage"
    ],
    "Resource": "*"
},

{
    "Sid": "RequiredIfUsingCustomKMSKeys",
    "Effect": "Allow",
    "Action": [
        "kms:ReEncrypt*",
        "kms:GenerateDataKey*",
        "kms:CreateGrant",
        "kms:DescribeKey*"
    ],
    "Resource": [
        "((kms_key_arn))"
    ]
},