Safety Violations experienced during an Operations Manager upgrade for DNS certificates
Article ID: 293762
Updated On:
Products
Issue/Introduction
Important Note #1
This guide is not for normal operations. Instead, it is for repairs to an upgrade issue experienced in versions 2.10.9+ of Ops Manager. To rotate in normal operations, please use the API Regenerate Non-Configurable rotation, details for this process are listed in the following documentation:
Note 1: Please verify the health of the deployment. Certificates will not update on VMs which are in an unhealthy state and as such, deployments will fail.
Note 2: Ignored VMs will not be updated. Make sure to unignore any VM you wish to update. This can cause deployments to list across multiple certificate version IDs.
How to prevent this issue from occurring in the upgrade
- Make sure you have completed all Root Certificate rotations and only have one Root CA. If you have two Root CA certificates in play (active:true/active:false) then complete the "delete" step of the "/api Root Certificate 3 round rotation", instructions can be found in the following link:
- http://OPSMANFQDN.com/api/v0/certificate_authorities
- Perform an All Tile Apply Change with "Upgrade service instance" errands to make sure all deployments are up to date.
Cause
During the Upgrade of Ops Manager to 2.10.9 or above, there is a configuration update to the Leaf certificates authored by the CA "/opsmgr/bosh_dns/tls_ca" . This configuration update is a modification to the "Subject Alternative Name", or SAN for short, list used by the Leaf certificates.
If a certificate rotation is not yet complete, it will error with the following:
TAS Step named: Regenerating BOSH DNS Certificates to Include SAN Will fail with: “Exited with 1"
On the OpsMan VM, the log at /var/log/opsmanager/production.log will feature the following:
/var/log/opsmanager/production.log:
[2021-05-03T18:18:02.756508 #813] ERROR -- : maestro CLI failed with "{
"safety_violations": [
{
"violation": "<safety violation>",
"certificate_names": [
"/opsmgr/bosh_dns/tls_ca"
]
}
],
"error": "safety constraints violated"
}", exit code 1
This is also found at https://OPSMAN-FQDN/debug/rails_log
Environment
Resolution
The solution is to complete the certificate rotation for the afflicted tree. This tree is for the CA "/opsmgr/bosh_dns/tls_ca" .
Usually this would be done with Apply Changes. If you are experiencing complications with doing so, you will have to perform "Manual deployments". We will be using a few different deployments in these examples, all Bosh Deployments should be able to use this method. If the manual method fails, investigate further into that specific Bosh deployment.
Manual Deployment Method:
export deployment_name="cf-77777777777777777777" bosh -d $deployment_name manifest > $deployment_name.yml chmod a+r $deployment_name.yml bosh -d $deployment_name deploy $deployment_name.yml
The next component you must know is that there are four leaf certificates under one CA. This is the tree we need to complete the rotation of:
topology: - name: /opsmgr/bosh_dns/tls_ca signs: - name: /bosh_dns_health_client_tls - name: /bosh_dns_health_server_tls - name: /dns_api_client_tls - name: /dns_api_server_tls
Pay close attention to the following fields:
topology: - name: CA certificate_id: Value versions: - version_id: Value deployment_names: (list of deployments utilizing this version) transitional: (True/False) signs: - name: Leaf certificate_id: signed_by: CA_version_id versions: - version_id: Value deployment_names: (list of deployments utilizing this version)
A guide to safety violations can be found at the following link:
Guide - Safety Violations and their corresponding Examples
The following safety violation can be experienced with either Example 1 or Example 3B (both examples are listed further along in this KB article):there is more than one signing version of a certificate authority
The following safety violation can be experienced with either Example 3, Example 3a, Example 3c, or Example 4 (all examples are listed further along in this KB article):
signing version has to be transitional if there is a transitional certificate authority
Important Note #2:
To use this troubleshooting tree, please find the example that most aligns with your current output.
Example 1
maestro topology --name /opsmgr/bosh_dns/tls_ca topology: - name: /opsmgr/bosh_dns/tls_ca certificate_id: 00000000-0000-0000-0000-000000000000 signed_by: /opsmgr/bosh_dns/tls_ca versions: - version_id: 33333333-3333-3333-3333-333333333333 transitional: true certificate_authority: true generated: true valid_until: 2025-05-04T22:59:48Z - version_id: 11111111-1111-1111-1111-111111111111 active: true deployment_names: - cf-77777777777777777777 - pivotal-mysql-8888888888888888888 - service-instance_99999999-9999-9999-9999-9999999999 signing: true certificate_authority: true generated: true valid_until: 2025-05-04T18:31:50Z signs: - name: /bosh_dns_health_client_tls certificate_id: f2c6caf2-ce55-412b-81da-3df6096e5c74 signed_by: /opsmgr/bosh_dns/tls_ca versions: - version_id: 22222222-2222-2222-2222-222222222222 active: true signed_by_version: 11111111-1111-1111-1111-111111111111 deployment_names: - cf-77777777777777777777 - pivotal-mysql-8888888888888888888 - service-instance_99999999-9999-9999-9999-9999999999 generated: true valid_until: 2022-05-05T18:31:51Z
To complete this version:
maestro update transitional signing --name /opsmgr/bosh_dns/tls_ca
You should now be at Example 2.
Note 3: Regarding Example 2, this is the command being run by the upgrade and is featured during the Apply Change as "Regenerating BOSH DNS Certificates to Include SAN".
Note 4: Regarding Example 2, if you have recently performed the /activate Ops Manager api call but have yet to perform the /regenerate api call and Apply Change with this upgrade, it will perform this call to the bosh_dns tree along with a few others. This has interfered with Root Certificate rotations at the /regenerate step. If this occurs, you can complete the afflicted trees in a similar manner.
Example 2
maestro topology --name /opsmgr/bosh_dns/tls_ca topology: - name: /opsmgr/bosh_dns/tls_ca certificate_id: 00000000-0000-0000-0000-000000000000 signed_by: /opsmgr/bosh_dns/tls_ca versions: - version_id: 33333333-3333-3333-3333-333333333333 active: true deployment_names: signing: true certificate_authority: true generated: true valid_until: 2025-05-04T22:59:48Z - version_id: 11111111-1111-1111-1111-111111111111 transitional: true active: true deployment_names: - cf-77777777777777777777 - pivotal-mysql-8888888888888888888 - service-instance_99999999-9999-9999-9999-9999999999 certificate_authority: true generated: true valid_until: 2025-05-04T18:31:50Z signs: - name: /bosh_dns_health_client_tls certificate_id: f2c6caf2-ce55-412b-81da-3df6096e5c74 signed_by: /opsmgr/bosh_dns/tls_ca versions: - version_id: 22222222-2222-2222-2222-222222222222 active: true signed_by_version: 11111111-1111-1111-1111-111111111111 deployment_names: - cf-77777777777777777777 - pivotal-mysql-8888888888888888888 - service-instance_99999999-9999-9999-9999-9999999999 generated: true valid_until: 2022-05-05T18:31:51Z
To complete this version:
maestro regenerate leaf --signed-by /opsmgr/bosh_dns/tls_ca
You should now be at Example 3.
Example 3
maestro topology --name /opsmgr/bosh_dns/tls_ca topology: - name: /opsmgr/bosh_dns/tls_ca certificate_id: 00000000-0000-0000-0000-000000000000 signed_by: /opsmgr/bosh_dns/tls_ca versions: - version_id: 33333333-3333-3333-3333-333333333333 active: true signing: true certificate_authority: true generated: true valid_until: 2025-05-04T22:59:48Z - version_id: 11111111-1111-1111-1111-111111111111 active: true signing: true deployment_names: - cf-77777777777777777777 - pivotal-mysql-8888888888888888888 - service-instance_99999999-9999-9999-9999-9999999999 transitional: true certificate_authority: true generated: true valid_until: 2025-05-04T18:31:50Z signs: - name: /bosh_dns_health_client_tls certificate_id: f2c6caf2-ce55-412b-81da-3df6096e5c74 signed_by: /opsmgr/bosh_dns/tls_ca versions: - version_id: 44444444-4444-4444-4444-444444444444 active: true signed_by_version: 33333333-3333-3333-3333-333333333333. <--- Signed by New Ca Version ID deployment_names: generated: true valid_until: 2022-05-05T22:59:48Z - version_id: 22222222-2222-2222-2222-222222222222 active: true signed_by_version: 11111111-1111-1111-1111-111111111111 <--- Signed by Old Ca Version ID deployment_names: - cf-77777777777777777777 - pivotal-mysql-8888888888888888888 - service-instance_99999999-9999-9999-9999-9999999999 generated: true valid_until: 2022-05-05T18:31:51Z
To complete this version, deploy every listed deployment via the Manual Deployment Method.
- Deployments under 1111 and 2222
You should now be at Example 4.
Important Note #3
This note applies if prior certificate rotations are incomplete or if a step is performed out of order prior to starting this procedure. In those cases, you might be observing one of the following alternate conditions. Please review the 3a, 3b, or 3c examples.
Please review the following prior to continuing to Example 4:
Example 3 alternative version A
maestro topology --name /opsmgr/bosh_dns/tls_ca topology: - name: /opsmgr/bosh_dns/tls_ca certificate_id: 00000000-0000-0000-0000-000000000000 signed_by: /opsmgr/bosh_dns/tls_ca versions: - version_id: 33333333-3333-3333-3333-333333333333 active: true deployment_names: - cf-77777777777777777777 - pivotal-mysql-8888888888888888888 - service-instance_99999999-9999-9999-9999-9999999999 signing: true certificate_authority: true generated: true valid_until: 2025-05-04T22:59:48Z - version_id: 11111111-1111-1111-1111-111111111111 transitional: true certificate_authority: true generated: true valid_until: 2025-05-04T18:31:50Z signs: - name: /bosh_dns_health_client_tls certificate_id: f2c6caf2-ce55-412b-81da-3df6096e5c74 signed_by: /opsmgr/bosh_dns/tls_ca versions: - version_id: 44444444-4444-4444-4444-444444444444 generated: true valid_until: 2022-05-05T22:59:48Z - version_id: 22222222-2222-2222-2222-222222222222 active: true signed_by_version: 33333333-3333-3333-3333-333333333333 deployment_names: - cf-77777777777777777777 - pivotal-mysql-8888888888888888888 - service-instance_99999999-9999-9999-9999-9999999999 generated: true valid_until: 2022-05-05T18:31:51Z
To complete this version, deploy every listed deployment via the Manual Deployment Method
- Deployments under 2222
You should now be at Example 4.
Example 3 alternative version B
maestro topology --name /opsmgr/bosh_dns/tls_ca topology: - name: /opsmgr/bosh_dns/tls_ca certificate_id: 00000000-0000-0000-0000-000000000000 signed_by: /opsmgr/bosh_dns/tls_ca versions: - version_id: 33333333-3333-3333-3333-333333333333 active: true deployment_names: - cf-77777777777777777777 - pivotal-mysql-8888888888888888888 signing: true certificate_authority: true generated: true valid_until: 2025-05-04T22:59:48Z - version_id: 11111111-1111-1111-1111-111111111111 active: true deployment_names: - service-instance_99999999-9999-9999-9999-9999999999 transitional: true signing: true certificate_authority: true generated: true valid_until: 2025-05-04T18:31:50Z signs: - name: /bosh_dns_health_client_tls certificate_id: f2c6caf2-ce55-412b-81da-3df6096e5c74 signed_by: /opsmgr/bosh_dns/tls_ca versions: - version_id: 44444444-4444-4444-4444-444444444444 active: true signed_by_version: 33333333-3333-3333-3333-333333333333 <---Different Version ID deployment_names: - cf-77777777777777777777 - pivotal-mysql-8888888888888888888 generated: true valid_until: 2022-05-05T22:59:48Z - version_id: 22222222-2222-2222-2222-222222222222 active: true signed_by_version: 11111111-1111-1111-1111-111111111111 <---Different Version ID deployment_names: - service-instance_99999999-9999-9999-9999-9999999999 generated: true valid_until: 2022-05-05T18:31:51Z
To complete this version, deploy oldest version deployments via the Manual Deployment Method
- Deployments under 1111 and 2222
You should now be at Example 4.
Example 3 alternative version C
maestro topology --name /opsmgr/bosh_dns/tls_ca topology: - name: /opsmgr/bosh_dns/tls_ca certificate_id: 00000000-0000-0000-0000-000000000000 signed_by: /opsmgr/bosh_dns/tls_ca versions: - version_id: 33333333-3333-3333-3333-333333333333 active: true deployment_names: - cf-77777777777777777777 - pivotal-mysql-8888888888888888888 - service-instance_99999999-9999-9999-9999-9999999999 signing: true certificate_authority: true generated: true valid_until: 2025-05-04T22:59:48Z - version_id: 11111111-1111-1111-1111-111111111111 transitional: true certificate_authority: true generated: true valid_until: 2025-05-04T18:31:50Z signs: - name: /bosh_dns_health_client_tls certificate_id: f2c6caf2-ce55-412b-81da-3df6096e5c74 signed_by: /opsmgr/bosh_dns/tls_ca versions: - version_id: 44444444-4444-4444-4444-444444444444 active: true signed_by_version: 33333333-3333-3333-3333-333333333333 <---Similar Version ID deployment_names: - cf-77777777777777777777 - pivotal-mysql-8888888888888888888 generated: true valid_until: 2022-05-05T22:59:48Z - version_id: 22222222-2222-2222-2222-222222222222 active: true signed_by_version: 33333333-3333-3333-3333-333333333333 <---Similar Version ID deployment_names: - service-instance_99999999-9999-9999-9999-9999999999 generated: true valid_until: 2022-05-05T22:59:48Z
To complete this version, follow the instructions below for either the Safe or Unsafe method.
Safe Version:
Deploy oldest version deployment via the Manual Deployment Method
- Deployments under 2222
Unsafe Version:
- Run command in Example 4 with " --skip-safety-validation" so that we can verify that all deployments have a new CA version even though the leaf version differs.
You should now be at Example 4
Example 4
maestro topology --name /opsmgr/bosh_dns/tls_ca topology: - name: /opsmgr/bosh_dns/tls_ca certificate_id: 00000000-0000-0000-0000-000000000000 signed_by: /opsmgr/bosh_dns/tls_ca versions: - version_id: 33333333-3333-3333-3333-333333333333 active: true deployment_names: - cf-77777777777777777777 - pivotal-mysql-8888888888888888888 - service-instance_99999999-9999-9999-9999-9999999999 signing: true certificate_authority: true generated: true valid_until: 2025-05-04T22:59:48Z - version_id: 11111111-1111-1111-1111-111111111111 transitional: true certificate_authority: true generated: true valid_until: 2025-05-04T18:31:50Z signs: - name: /bosh_dns_health_client_tls certificate_id: f2c6caf2-ce55-412b-81da-3df6096e5c74 signed_by: /opsmgr/bosh_dns/tls_ca versions: - version_id: 44444444-4444-4444-4444-444444444444 active: true signed_by_version: 33333333-3333-3333-3333-333333333333 deployment_names: - cf-77777777777777777777 - pivotal-mysql-8888888888888888888 - service-instance_99999999-9999-9999-9999-9999999999 generated: true valid_until: 2022-05-05T22:59:48Z - version_id: 22222222-2222-2222-2222-222222222222 generated: true valid_until: 2022-05-05T18:31:51Z
To complete this version:
maestro update-transitional remove --name /opsmgr/bosh_dns/tls_ca
Apply Change should now succeed.
Feedback
