How to secure EC2 user data for BOSH VMs on AWS

Products

• Operations Manager

Issue/Introduction

For security issue, customer don't want the sensitive data visible in EC2 user data. So they would like to know how to secure sensitive information in EC2 user data. Below is the example of Diego_cell instance's EC2 user data in AWS console:

Environment

Product Version: 2.10

Resolution

Enclosed is a list of credentials maintained in the user-data field.
Credentials in User-Data:

Key	Credential Type
blobstore.options.access_key_id	ID
blobstore.options.secret_access_key	Key
blobstore.options.password	Password
env.bosh.password	Password
env.bosh.mbus.cert.ca	Root CA
env.bosh.mbus.cert.private_key	Private Key
env.bosh.mbus.cert.certificate	Certificate
env.bosh.blobstores.options.tls.cert.ca	Root CA
env.bosh.blobstores.options.password	Password

 

These credentials map to the respective system components:

Key	System Component
blobstore.options.access_key_id	Blobstore
blobstore.options.secret_access_key	Blobstore
blobstore.options.password	Blobstore
env.bosh.password	BOSH vcap user
env.bosh.mbus.cert.ca	NATS Messaging System
env.bosh.mbus.cert.private_key	NATS Messaging System
env.bosh.mbus.cert.certificate	NATS Messaging System
env.bosh.blobstores.options.tls.cert.ca	Blobstore
env.bosh.blobstores.options.password	Blobstore

 

General Recommendations

• We advise customers to use IAM Profile over Static Credentials(Access Keys and Secrets) when given the choice. We advise customers to move to signed URLs in the blobstore. You can find some of the docs for the feature here .
• Ops Manager 3.0.5 shipped with a short lived NATS certificates to secure credentials associated with NATS communication. The credentials will still be in the metadata info, but they will be unusable by the time workloads are started on the VM. Bosh deploys the VM, and when it connects back to the director with the first set of credentials, it issues the VM new creds and deactivates the old ones. You can find some of the docs for the feature here.
• We recommend customers to upgrade to the latest version of Ops Manager, as 3.0.5 and several older releases have bugs and security issues that have been fixed in later patches.

How to Guide
1) For blobstore.options.access_key_id, it can be secured by Enable signed URLs in Director Config --> Blobstore Location and all bosh VMs will need to be recreated (e.g, if you've already deployed with the SignedURLs unselected, you'd need to tick this box and ALSO check the Director Config --> Recreate VMs deployed by the BOSH Director).
Note: Only major requirement is stemcells need to at least be xenial 621 line.

2) For securing env.bosh.mbus.cert.private_key and env.bosh.mbus.cert.certificate, Ops Manager 3.0.5 shipped with a short lived NATS certificates to solve this problem. This feature has been released on Operations Manager 3.0.5 and it will affect only newly created/recreated VMs. If you want to configure VMs in existing Tanzu Operations Manager deployments to use short-lived bootstrap credentials, you must recreate them.
Note: When an installation is using this feature it is a MUST to ensure that the stemcells used in the director are compatible with it; stemcells that are incompatible will result in unresponsive VMs. The oldest compatible stemcell versions are:

• Windows 2019 - 2019.41 and later
• Ubuntu Xenial - 621.171 and later
• Ubuntu Bionic - 1.36 and later
• Ubuntu Jammy - 1.95 and later