How to secure EC2 user data for BOSH VMs on AWS
Article ID: 293813
Updated On:
Products
Issue/Introduction
For security issue, customer don't want the sensitive data visible in EC2 user data. So they would like to know how to secure sensitive information in EC2 user data. Below is the example of Diego_cell instance's EC2 user data in AWS console:
Environment
Product Version: 2.10
Resolution
Enclosed is a list of credentials maintained in the user-data field.
Credentials in User-Data:
| Key | Credential Type |
|---|---|
blobstore.options.access_key_id | ID |
blobstore.options.secret_access_key | Key |
blobstore.options.password | Password |
env.bosh.password | Password |
env.bosh.mbus.cert.ca | Root CA |
env.bosh.mbus.cert.private_key | Private Key |
env.bosh.mbus.cert.certificate | Certificate |
env.bosh.blobstores.options.tls.cert.ca | Root CA |
env.bosh.blobstores.options.password | Password |
These credentials map to the respective system components:
| Key | System Component |
|---|---|
blobstore.options.access_key_id | Blobstore |
blobstore.options.secret_access_key | Blobstore |
blobstore.options.password | Blobstore |
env.bosh.password | BOSH vcap user |
env.bosh.mbus.cert.ca | NATS Messaging System |
env.bosh.mbus.cert.private_key | NATS Messaging System |
env.bosh.mbus.cert.certificate | NATS Messaging System |
env.bosh.blobstores.options.tls.cert.ca | Blobstore |
env.bosh.blobstores.options.password | Blobstore |
General Recommendations
- We advise customers to use IAM Profile over Static Credentials(Access Keys and Secrets) when given the choice. We advise customers to move to signed URLs in the blobstore. You can find some of the docs for the feature here .
- Ops Manager 3.0.5 shipped with a short lived NATS certificates to secure credentials associated with NATS communication. The credentials will still be in the metadata info, but they will be unusable by the time workloads are started on the VM. Bosh deploys the VM, and when it connects back to the director with the first set of credentials, it issues the VM new creds and deactivates the old ones. You can find some of the docs for the feature here.
- We recommend customers to upgrade to the latest version of Ops Manager, as 3.0.5 and several older releases have bugs and security issues that have been fixed in later patches.
How to Guide
1) For blobstore.options.access_key_id, it can be secured by Enable signed URLs in Director Config --> Blobstore Location and all bosh VMs will need to be recreated (e.g, if you've already deployed with the SignedURLs unselected, you'd need to tick this box and ALSO check the Director Config --> Recreate VMs deployed by the BOSH Director).
Note: Only major requirement is stemcells need to at least be xenial 621 line.
2) For securing env.bosh.mbus.cert.private_key and env.bosh.mbus.cert.certificate, Ops Manager 3.0.5 shipped with a short lived NATS certificates to solve this problem. This feature has been released on Operations Manager 3.0.5 and it will affect only newly created/recreated VMs. If you want to configure VMs in existing Tanzu Operations Manager deployments to use short-lived bootstrap credentials, you must recreate them.
Note: When an installation is using this feature it is a MUST to ensure that the stemcells used in the director are compatible with it; stemcells that are incompatible will result in unresponsive VMs. The oldest compatible stemcell versions are:
- Windows 2019 - 2019.41 and later
- Ubuntu Xenial - 621.171 and later
- Ubuntu Bionic - 1.36 and later
- Ubuntu Jammy - 1.95 and later
Feedback
