Rotating /services/tls_ca certificate using CredHub transitional certificate rotation feature
Article ID: 293894
Updated On:
Products
Issue/Introduction
IMPORTANT: Please do not attempt this procedure if your /services/tls_ca is already expired. If it has already expired, please see the Knowledge Base (KB) article, How to rotate an already expired /services/tls_ca certificate.
This procedure is only applicable to Operations Manager 2.8 or later versions. The reason this procedure can not be applied to Operations Manager 2.6 or older is because BOSH Director CredHub does not support CA certificate concatenation in the older release.
Additionally, you should only need this procedure when you have:
- A MySQL 2.x tile with TLS enabled
- Bound apps that do not source their CA certificates from the jdbcURL or "/etc/ssl/certs/ca-certificate.crt". Only apps that use the VCAP_SERVICES "ca" variable, as per below, will need to follow this procedure to avoid downtime for those apps.
This procedure is also compatible with VMware Tanzu GemFire, VMware Tanzu for Redis, and VMware Tanzu for RabbitMQ.
"p.mysql":
"credentials": {
"hostname": "q-n2s0.q-g674.bosh",
"jdbcUrl": "jdbc:mysql://q-n2s0.q-g674.bosh:3306/service_instance_db?user=<usename>&password=<password>&useSSL=true&requireSSL=true&serverSslCert=/etc/ssl/certs/ca-certificates.crt",
"tls": {
"cert": {
"ca": "-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----\n"
The official procedure to rotate /services/tls_ca can be found in the documentation linked below.
Rotate the Services TLS CA and its leaf certificates: https://techdocs.
Resolution
Restart Points
This procedure uses environment variables which allow you to simply copy and paste commands into your terminal. Throughout this procedure your terminal session might expire, requiring you to recreate the environment variables set along the way. The procedure is marked with "Restart Points" which are reminders to recreate the environment variables before proceeding. Here are the commands you need to run in order to ensure you have the environment variables set.
You do not need to run these commands unless you have reached a "Restart Point".
export SERVICES_TLS_CA_ID=`credhub curl -p /api/v1/certificates?name=%2Fservices%2Ftls_ca | jq -r '.certificates[]| .id'` export ORIG_SERVICES_TLS_CA_VERSION=`credhub curl -p /api/v1/certificates?name=%2Fservices%2Ftls_ca | jq -r '.certificates[] | .versions | sort_by(.expiry_date) | .[0].id '` export NEW_SERVICES_TLS_CA_VERSION=`credhub curl -p /api/v1/certificates?name=%2Fservices%2Ftls_ca | jq -r '.certificates[] | .versions | sort_by(.expiry_date) | .[1].id '`
IMPORTANT
This procedure assumes you only have one version of the /service/tls_ca created in CredHub. The below command will print out how many versions of the /services/tls_ca you currently have. If you see more than one version, please delete any erroneous versions which have the "transitional" flag set to true. You should only have one active version which has "transitional" flag set as false.
Note: CredHub CLI needs to be authenticated before running the below command. Please see Connecting to Credhub for tips on how to authenticate with the BOSH Credhub instance.
credhub curl -p /api/v1/certificates?name=%2Fservices%2Ftls_ca
Connecting to CredHub
1. Get BOSH environment variables from Operations Manager and apply them to your terminal session. The follow is an example Ops Manager URL: https://<opsman.hostname>/api/v0/deployed/director/credentials/bosh_commandline_credentials.
The below export command should be derived from the Operations Manager values:
export BOSH_CLIENT=ops_manager BOSH_CLIENT_SECRET=<value from opsman> BOSH_CA_CERT=/var/tempest/workspaces/default/root_ca_certificate BOSH_ENVIRONMENT=<value from opsman>
Set CredHub environment variables
As long as the BOSH environment variables are set, you can just copy and paste this command into your terminal without modification.
export CREDHUB_SERVER=https://${BOSH_ENVIRONMENT}:8844 CREDHUB_CLIENT=${BOSH_CLIENT} CREDHUB_SECRET=${BOSH_CLIENT_SECRET} CREDHUB_CA_CERT=${BOSH_CA_CERT}
Fetch the /services/tls_ca cert id from CredHub
export SERVICES_TLS_CA_ID=`credhub curl -p /api/v1/certificates?name=%2Fservices%2Ftls_ca | jq -r '.certificates[]| .id'`; echo $SERVICES_TLS_CA_ID
Regenerate the /services/tls_ca
Note: If you run this command multiple times it could generate multiple versions of the certificate. This procedure will not work if there are multiple versions. If you do make this mistake, please make sure to remove all erroneous versions which are not the original cert and the latest version before proceeding. Refer to the step "Delete old version of /services/tls_ca from CredHub" in this procedure for an example of how to delete a version of /service/tls_ca.
export NEW_SERVICES_TLS_CA_VERSION=`credhub curl -p /api/v1/certificates/$SERVICES_TLS_CA_ID/regenerate -X=POST -d '{"set_as_transitional": true}' | jq -r '.id'`; echo $NEW_SERVICES_TLS_CA_VERSION
Verify you have a cert with two versions
credhub curl -p /api/v1/certificates?name=%2Fservices%2Ftls_ca
The output should look similar to the output below which shows the original CA cert with transitional set as false and the new CA cert with transitional set as true.
{
"certificates": [
{
"id": "9c27fa2e-4d32-4593-a3e8-07dd5e1161ef",
"name": "/services/tls_ca",
"signed_by": "/services/tls_ca",
"signs": [
"/p-bosh/service-instance_########-####-####-####-############/gemfire-locator-certificate",
"/p-bosh/service-instance_########-####-####-####-############/gemfire-server-certificate",
"/p-bosh/service-instance_########-####-####-####-############/agent_client_tls",
"/p-bosh/service-instance_########-####-####-####-############/agent_server_tls",
"/p-bosh/service-instance_########-####-####-####-############/mysql_server_tls",
"/p-bosh/service-instance_########-####-####-####-############/gemfire-locator-certificate",
"/p-bosh/service-instance_########-####-####-####-############/gemfire-server-certificate",
"/p-bosh/service-instance_########-####-####-####-############/tls_certificate",
"/p-bosh/service-instance_########-####-####-####-############/redis_certificate",
"/p-bosh/service-instance_########-####-####-####-############/gemfire-locator-certificate",
"/p-bosh/service-instance_########-####-####-####-############/gemfire-server-certificate",
"/p-bosh/service-instance_########-####-####-####-############/gemfire-locator-certificate",
"/p-bosh/service-instance_########-####-####-####-############/gemfire-server-certificate",
"/p-bosh/service-instance_########-####-####-####-############/gemfire-locator-certificate",
"/p-bosh/service-instance_########-####-####-####-############/gemfire-server-certificate",
"/p-bosh/service-instance_########-####-####-####-############/tls_certificate"
],
"versions": [
{
"certificate_authority": true,
"expiry_date": "2025-06-03T14:37:49Z",
"generated": true,
"id": "504d53db-cd7a-4e19-82f8-046fa429278b",
"self_signed": true,
"transitional": true
},
{
"certificate_authority": true,
"expiry_date": "2025-05-23T00:20:13Z",
"generated": true,
"id": "9717ef4e-fa29-4f06-b3c0-97f25f4a85e6",
"self_signed": true,
"transitional": false
}
]
}
]
}
Set the cert version environment variables
export ORIG_SERVICES_TLS_CA_VERSION=`credhub curl -p /api/v1/certificates?name=%2Fservices%2Ftls_ca | jq -r '.certificates[] | .versions | sort_by(.expiry_date) | .[0].id '`; echo $ORIG_SERVICES_TLS_CA_VERSION
export NEW_SERVICES_TLS_CA_VERSION=`credhub curl -p /api/v1/certificates?name=%2Fservices%2Ftls_ca | jq -r '.certificates[] | .versions | sort_by(.expiry_date) | .[1].id '`; echo $NEW_SERVICES_TLS_CA_VERSION
Update Director Tile
Get the new version of /services/tls_ca certificate. This should output the new CA cert you can copy into various Operations Manager fields.
credhub curl -p /api/v1/certificates/$SERVICES_TLS_CA_ID/versions | jq -r ".[]| select(.id==\"$NEW_SERVICES_TLS_CA_VERSION\") | .value.ca" | sed 's/\\n/\n/g'
- Navigate to the Installation Dashboard in Ops Manager and click the BOSH Director tile.
- Click Security
- Append the new /services/tls_ca certificate to the Trusted Certificates Field
- Click Save
Update TAS and Isolation Segment tile with new Cert
- Make sure both the old and the new CA values are pasted into TAS for VMs tile > Networking > Certificate Authorities Trusted by GoRouter
- Make sure both the old and the new CA values are pasted into TAS for VMs tile > Networking > Certificate Authorities Trusted by HAProxy
Apply Changes #1
- Ensure All tiles are checked, especially TAS and Isolation Segments.
- Ensure All On Demand Service tiles such as MySQL, VMware Tanzu for RabbitMQ, VMware Tanzu for Redis, and VMware Tanzu GemFire have "Upgrade all service instances" errand checked.
--------------------------------- RESTART POINT --------------------------------------
Given the amount of time apply changes will take it is likely your terminal session will be closed. Please refer to the "Restart Points" section for a list of environment variables you can set before proceeding.
------------------------------------------------------------------------------------------------
Rebind MySQL Apps
After the first Apply Changes completes successfully you may or may not need to rebind all MySQL Applications. As previously discussed in the beginning of this article, only certain applications will need to rebind. Here is a helpful way to identify which applications are using MySQL on demand instances.
It can be helpful to identify exactly which are using TLS enabled MySQL On Demand Instances. Get a list of TLS Enabled service instances:
credhub curl -p /api/v1/certificates?name=%2Fservices%2Ftls_ca | jq -r '.certificates[] | .signs[]' | egrep mysql_server_tls
Get applications bound to specified service deployments. After replacing "cf-<DEPLOYMENT_GUID>" in the below command with your cf deployment. This command will return the apps that use these services.
bosh -d cf-<DEPLOYMENT_GUID> ssh mysql/0 -c "sudo mysql --defaults-file=/var/vcap/jobs/pxc-mysql/config/mylogin.cnf --batch -e \"use ccdb; SELECT services.label AS service_label ,service_plans.name AS service_plan, organizations.name AS org_name, spaces.name AS space_name, service_instances.guid AS service_instance_guid, service_instances.name AS service_instance_name, apps.name AS app_name FROM service_instances JOIN service_plans ON (service_instances.service_plan_id = service_plans.id) JOIN services ON (service_plans.service_id = services.id) LEFT JOIN service_bindings ON (service_bindings.service_instance_guid = service_instances.guid) LEFT JOIN apps ON (service_bindings.app_guid = apps.guid) JOIN spaces ON (service_instances.space_id = spaces.id) JOIN organizations ON (spaces.organization_id = organizations.id) WHERE services.label = 'p.mysql';\"" --column=Stdout --results
Rebind or re-stage the identified application:
cf unbind-service APP-NAME SERVICE-NAME cf bind-service APP-NAME SERVICE-NAME cf restage APP-NAME
Set the old /services/tls_ca certificate as transitional. After running this command the new CA will be the signing cert going forward.
credhub curl -p /api/v1/certificates/$SERVICES_TLS_CA_ID/update_transitional_version -X=PUT -d "{\"version\": \"$ORIG_SERVICES_TLS_CA_VERSION\"}"Regenerate leaf certificates:
This step is only required for service tiles that do not set manifest property "update_mode: converge" for their leaf certificate variables. Most newer release of tiles already include this property but some of the older releases may not. To avoid leaf certs getting missed it is safer to manually trigger the leaf cert regeneration.
credhub bulk-regenerate --signed-by /services/tls_ca
Apply Changes #2
- The reason why we do not recommend applying changes to TAS or other unrelated tiles is to help make this procedure go faster.
- Do not deploy changes to TAS. Ensure TAS is not checked.
- Do not deploy changes to Isolation Segments. Ensure Isolations segments are not checked.
- Only deploy changes to On Demand Service Tiles such as MySQL, VMware Tanzu for RabbitMQ, VMware Tanzu for Redis, and VMware GemFire Tanzu.
- Ensure "Upgrade all service instances" errand is checked for all on demand service tiles.
Note: This apply changes will trigger the leaf certificates to be rotated and signed by the new /services/tls_ca version.
--------------------------------- RESTART POINT --------------------------------------
Given the amount of time apply changes will take it is likely your terminal session will be closed. Please refer to the "Restart Points" section for a list of environment variables you can set before proceeding.
------------------------------------------------------------------------------------------------
Remove old /services/tls_ca from BOSH Director tile
Now your apps have reconnected to service instances with certificates generated by the new CA, remove the old CA certificate:
- Navigate to the installation dashboard in Operations Manager and click the BOSH Director tile
- Click Security
- Delete the old /services/tls_ca certificate from the Trusted Certificates Field.
- Click Save
- Update TAS and all Isolation Segment tiles with the following changes
- Delete the old /services./tls_ca certificate from Networking > Certificate Authorities Trusted by GoRouter.
- Delete the old /services/tls_ca certificate from Networking > Certificate Authorities Trusted by HAProxy.
Delete the old /services/tls_ca from Credhub
credhub curl -p /api/v1/certificates/$SERVICES_TLS_CA_ID/versions/$ORIG_SERVICES_TLS_CA_VERSION -X=DELETE
Apply Changes #3
Note: This step does not require immediate action. You can choose to apply changes now or wait to do so at a future date of your choosing. It is a good practice to remove outdated certificates as soon as possible. Also if your apps use OpenSSL 1.1.0 to build their certificate chains then please be aware of the KB article, Cloud Controller fails to create service instance when there is a expired cert installed on the system.
- Ensure All tiles are checked, especially TAS and Isolation Segments
- Ensure All On Demand Service tiles such as MySQL, VMware Tanzu for RabbitMQ, VMware Tanzu for Redis, and VMware Tanzu GemFire have "Upgrade all service instances" errand checked.
Feedback
