Applying new stemcell fails on AWS due to restrict AMI usage

Products

• Operations Manager

Issue/Introduction

When deploying new stemcell to TAS or TKGI on AWS, the BOSH deployment could fail with the following error.
 

Task 564263 | 01:00:02 | Compiling packages: golang-1-linux/e07fa80a3e9eb2c0beae3cad38ac914ffa7d56695458b672fa6ee126f2026145 (00:00:09)
L Error: Unknown CPI error 'Unknown' with message 'You are not authorized to perform this operation. Encoded authorization failure message: jxCkX1JPwLhrzGWXkn05QprmRYj-Xr8fv1gNUIIlWbwpcLsCQW673Lvk5s9TND-ceHQ-Op1qWH7dkRaRrqYfIfvsy4AiNhlaUilAWZi4Kjukcw2rZgZYvzSXTEpyTaSS5vadYizRKB4Qdt90ZtJsaeV2r7aWX-luqEFw6WyJrqAUtc5SSQunftVqwGfeX7ZvDtbYNzcE5wq2tRLLLRMp-RCDAiogVV3TpMumr0XDWR2fuhNk8Vrs6yoJ3BrV5nFpxvyWZdWiAxmPkXxU0rJPh9uUfTZ-UOiAEpI3vRsoawxZV36U8aZUyVABmbE_3t9CQatJwPOY9lg2Ha32LTuVdRSUBSRZW2Q9JK6gIpHBlV8bBLg91gIaBaEsOGXO2HOozo1zET9u2iI6aFROHTYR9nW774hvcIe44PG2CEaUVtM62tHaG1HATboIaOequQCsQT2b8yNTW5jW1H0QKOvz26tQO6MH8FNCiFCDW4CsUAXXDfG8ccPpEsQH86rsSLd-0vhwp5Kvo7V1UQUFF1euwkKhzK-lxvLymVMYWjDZORaOHexFP-tCDa_k1GDhnotKI2TYCfQAdwCcGWWVHem9dPRqiL_tXFDuOrHpWCHKBbgRu1CsiVvG8BHoV_3r_2hnd8cGP1squAGy3Ea80fQy_aoaoiOp8gvV-ILb7FQsUCUlU1UL3dC-IHPejz_81HJYUUZIOTV_papS7Eo6z-DOpKsWa9u_UVjBBZ4IX_qUEmdKYXWbFAXq9FjYhLzUgENRzigXnkFAYZJyuo503EyiDMaOUYh_koRSR_bq3LiLbk3d9lwloO7bv2Fkye3iYC4' in 'create_vm' CPI method (CPI request ID: 'cpi-764339')

From the error, we understand the BOSH CPI method 'create_vm' failed but the reason is hidden in the encoded authorization failure message returned by AWS.

Environment

Product Version: 2.9

Resolution

Steps for troubleshooting the issue.

1. Decode the encoded authorization failure message per AWS docs as follows.
   • AWS CLI doc: https://docs.aws.amazon.com/cli/latest/reference/sts/decode-authorization-message.html
   • Example AWS knowledge article: https://aws.amazon.com/premiumsupport/knowledge-center/aws-backup-encoded-authorization-failure/ 
2. Based on the decoded error, adjust your AWS IAM policy to allow the denied request. In the above example, the decoded error is as follows.

$ aws sts decode-authorization-message --encoded-message <encoded-message>

{
  "allowed": false,
  "explicitDeny": true,
  "matchedStatements": {
    "items": [
      {
        "statementId": "RestrictAMI",
        "effect": "DENY",
        "principals": {
          "items": [
            {
              "value": "AAAAABBBBBCCCCCDDDDDE"
            }
          ]
        },
        "principalGroups": {
          "items": []
        },
        "actions": {
          "items": [
            {
              "value": "ec2:RunInstances"
            }
          ]
        },
        "resources": {
          "items": [
            {
              "value": "arn:aws:ec2:*::image/*"
            }
          ]
        },
        "conditions": {
          "items": [
            {
              "key": "ec2:Owner",
              "values": {
                "items": [
                  {
                    "value": "934467097824"
                  },
                  {
                    "value": "085178109370"
                  },
                  {
                    "value": "364390758643"
                  }
                ]
              }
            },
            {
              "key": "ec2:ResourceTag/AllowAMI",
              "values": {
                "items": [
                  {
                    "value": "true"
                  }
                ]
              }
            }
          ]
        }
      }
    ]
  },
  "failures": {
    "items": []
  },
  "context": {
    "principal": {
      "id": "AAAAABBBBBCCCCCDDDDDE",
      "name": "pcf_iam_user",
      "arn": "arn:aws:iam::111112222233:user/system/pcf_iam_user"
    },
    "action": "ec2:RunInstances",
    "resource": "arn:aws:ec2:ap-southeast-2::image/ami-0d835c7a02a6f4fd3",
    "conditions": {
      "items": [
        {
          "key": "ec2:ImageID",
          "values": {
            "items": [
              {
                "value": "ami-0d835c7a02a6f4fd3"
              }
            ]
          }
        },
        {
          "key": "ec2:ImageType",
          "values": {
            "items": [
              {
                "value": "machine"
              }
            ]
          }
        },
        {
          "key": "aws:Resource",
          "values": {
            "items": [
              {
                "value": "image/ami-0d835c7a02a6f4fd3"
              }
            ]
          }
        },
        {
          "key": "aws:Account",
          "values": {
            "items": [
              {
                "value": "462397596885"
              }
            ]
          }
        },
        {
          "key": "ec2:IsLaunchTemplateResource",
          "values": {
            "items": [
              {
                "value": "false"
              }
            ]
          }
        },
        {
          "key": "ec2:RootDeviceType",
          "values": {
            "items": [
              {
                "value": "ebs"
              }
            ]
          }
        },
        {
          "key": "aws:Region",
          "values": {
            "items": [
              {
                "value": "ap-southeast-2"
              }
            ]
          }
        },
        {
          "key": "aws:Service",
          "values": {
            "items": [
              {
                "value": "ec2"
              }
            ]
          }
        },
        {
          "key": "ec2:Owner",
          "values": {
            "items": [
              {
                "value": "462397596885"
              }
            ]
          }
        },
        {
          "key": "ec2:Public",
          "values": {
            "items": [
              {
                "value": "true"
              }
            ]
          }
        },
        {
          "key": "aws:Type",
          "values": {
            "items": [
              {
                "value": "image"
              }
            ]
          }
        },
        {
          "key": "ec2:Region",
          "values": {
            "items": [
              {
                "value": "ap-southeast-2"
              }
            ]
          }
        },
        {
          "key": "aws:ARN",
          "values": {
            "items": [
              {
                "value": "arn:aws:ec2:ap-southeast-2::image/ami-0d835c7a02a6f4fd3"
              }
            ]
          }
        }
      ]
    }
  }
}

• The owner account (462397596885) of the new stemcell image (ami-0d835c7a02a6f4fd3) was not whitelisted in the policy statement "RestrictAMI", therefore the request to run instance based on this AMI was denied. The error is resolved by modifying the policy to whitelist the new owner account.
• To prevent such error, we suggest to whitelist the AMI owner of the new stemcell before applying it to TAS/TKGI on AWS. Below is the command to check owner account of an AMI.

$ aws --region ap-southeast-2  ec2 describe-images --image-ids ami-0d835c7a02a6f4fd3 | jq -r '.Images[] | "AMI: \(.ImageId), Owner: \(.OwnerId)"'
AMI: ami-0d835c7a02a6f4fd3, Owner: 462397596885

​​​​​​