Understanding and Rotating Single CA certificates using Maestro in Ops Manager
Article ID: 293816
Updated On:
Products
Issue/Introduction
Note: Utilizing this tool for the rotation of all certificates managed by Ops Manager is not supported. Instead, refer to Rotating CAs and Leaf Certificates. This guide can be used to repair any Certificate Trees that generate an error for safety violations experienced during normal rotations or remaining afterwards.
During normal operation of the platform, it is sometimes required to repair a single CA Certificate Tree. This can be for a few different reasons such as the certificate not rotating during normal operations or due to a safety violation. Safety violations are being presented by the "Maestro" utility, which is available on the Ops Manager VM. We can utilize this utility to diagnose the issue and implement / verify repairs. The safety violations each have corresponding troubleshooting steps but sometimes they can be confusing. For more informaiton about these steps, refer to Troubleshooting CredHub Maestro Safety Violations During Certificate Rotation. The reasons for these certificates being in a state may vary, but often they are due to incomplete certificate rotations or issues experienced during an upgrade. Some Certificate Trees are rotated completely independently.
This guide can help with identifying what specifically needs to be done by finding where the problematic Certificate Tree is and completing the rotation from there.
The organization of the platform is such that all CA's will reside under the Root CA and rotate along with it. The following diagram will help you visualize this:
Main Ops Manager Root CA Certificate Tree:
ROOT CA + | | +-------------------------------------------------------+ | | | | | | | | | + + + CA1 CA2 CA3 + + + | | | | | | +----+---------+---+ +----+---------+---+ +----+---------+---+ | | | | | | | | | | | | | | | + + + + + + + + + + + + + + + A B C D E F G H I J K L M N O
We can, however, separate out a single CA Tree and focus on just that. This diagram would be for just 1 of the CA's.
Single CA Certificate Tree:
CA
+
|
|
+----+---------+---+
| | | | |
+ + + + +
A B C D E
The Maestro utility provides a command called "Topology". These outputs will help show the Certificate and which Deployments are affected. All of these examples are outputs of the command:
maestro topology --name "/CA"
Environment
Resolution
Maestro's single CA rotation procedure is found in the following Maestro documentation: Rotate a Single CA and Its Leaf Certificates
This article can be used as a guide to help understand what to expect each state to be, and if you start at the middle, what to do next.
The "Transitional Rotation" is utilized for the "CA" portion of a tree and can be split into three rounds:
Round 1
CA has a new version "regenerated" automatically in a transitional state. At this point, it is not the "signing" CA.
Round 2
CA has the transitional flag swapped to the older version. The new version is marked as the "signing" CA. Leaf certificates are "regenerated" off of the new "signing" CA
Round 3
CA has transitional flag removed from the older version.
Each round is completed with an Apply Changes on the Deployments listed.
Single CA Full Transitional Rotation
The starting example shows one version of the /CA and the /Leaf. You may have multiple versions and multiple leafs.
Example 0:
/CA: - Version 1 - Deployments signs: - /Leaf 1 - Version 1 - Deployments
Round 1
We only have one active version of the CA and one active version of the leaf. We can now run the first command, the start of Round 1:
maestro regenerate ca --name /CA
Example 1:
/CA: - Version 2 - transitional: true - Version 1 - Deployments signs: - /Leaf 1 - Version 1 - Deployments - Signed by CA Version 1
After having created the new CA version, the list is updated to show "Version 2" which is in a transitional state. In Ops Manager 2.10+, this is automatically done. In 2.8 / 2.9, there is an additional command to mark the transitional version. We need to now populate this out to all of the Deployments listed in Version 1 of the /CA.
Apply Change on deployments listed in CA/Leaf Version 1. If the deployment includes service instance tile make sure to utilize "upgrade" errand.
Example 2:
/CA: - Version 2 - transitional:true - Version 1 - Deployments - signing: true signs: - /Leaf 1 - Version 1 - Deployments - Signed by CA Version 1
Round 2
Now that the /CA Version 2 has been distributed to all the deployments, we can swap the "transitional" flag to the /CA Version 1.
maestro update-transitional signing --name /CA
Example 3:
/CA: - Version 2 - signing: true - Version 1 - Deployments - transitional:true signs: - /Leaf 1 - Version 1 - Deployments - Signed by CA Version 1
By swapping the "transitional" flag to /CA Version 1, we allow /CA Version 2 to become the "signing" version. The second half of round 2 is to create a new leaf certificate version.
maestro regenerate leaf --signed-by "/CA"
Example 4:
/CA: - Version 2 - valid_until: date C - Version 1 - Deployments - transitional:true - signing: true - valid_until: date A signs: - /Leaf 1 - Version 2 - valid_until: date D - Version 1 - Deployments - Signed by CA Version 1 - valid_until: date B
*Note - Valid Until date is utilized here for certificate distinction as "signing:true" and "signed_by_version" are not present until after the deployment consumes the newest leaf version (Version 2 in this example).
At this stage, we have now a new Leaf (Version 2) that we need to deploy out to all the "deployments" under Version 1.
Apply Change on deployments listed in CA/Leaf Version 1 deployments. If the deployment includes service instance tile make sure to utilize "upgrade" errand.
Example 5:
/CA: - Version 2 - signing: true - Deployments - Version 1 - transitional:true signs: - /Leaf 1 - Version 2 - Deployments - Signed by CA Version 2 - Version 1
Round 3
Now that we have populated the New CA and the New Leaf versions out to the deployments that utilize them, we need to complete the rotation. This is done by removing the transitional flag.
maestro update-transitional remove --name /CA
Example 6:
/CA: - Version 2 - signing: true - Deployments - Version 1 signs: - /Leaf 1 - Version 2 - Deployments - Signed by CA Version 2 - Version 1
By removing the transitional flag we have completed the rotation commands and need to populate it out.
Apply Change on tiles listed in Version 2 deployments. If the deployment includes service instance tile make sure to utilize "upgrade" errand.
The Out Of Sync scenario
There are occasional circumstances where the leaf certificates don't all regenerate with the creation of a new CA version. This can cause the tree to become "Out of Sync" and requires manual intervention.
Example 7:
/CA: - Version 2 - signing: true - Deployments - Version 1 - signing: true - transitional: true signs: - /Leaf 1 = Version 1 - Deployments - Signed by CA Version 2 - /Leaf 2 - Version 1 - Deployments - Signed by CA Version 1
We have 2 signing CA's which will cause Maestro to throw a safety violation. We need to bring the leaf 2 under the newer CA version (/CA Version 2). We do this by regenerating the leaf so it will be signed by the newer CA version. Due to the safety violation, we will have to tell it to ignore this safety violation.
maestro regenerate leaf --name Leaf 2 --skip-safety-check
Example 8:
/CA: - Version 2 - signing: true - Deployments - Version 1 - signing: true - Deployments signs: - Leaf 1 = Version 1 - Deployments - Signed by CA Version 2 - Leaf 2 - Version 2 - Signed by CA Version 2 - Version 1 - Deployments - Signed by CA Version 1
There is now a second version of the /Leaf 2 certificate that we need to populate out.
Apply Change on tiles listed in /Leaf 2 Version 1 deployments. If the deployment includes service instance tile make sure to utilize "upgrade" errand.
Example 9:
/CA:
- Version 2
- signing: true
- Deployments
- Version 1
signs:
- /Leaf 1
- Version 1
- Deployments
- Signed by CA Version 2
- /Leaf 2
- Version 2
- Signed by CA Version 2
- Deployments
- Version 1
You should now have a topology that is more normal. If there is a transitional flag in play, pick it up at the start of round 3.
Single CA Leaf Only Certificate Rotation
If the CA portion of the tree is not going to expire soon but you wish to rotate the \leaf certificate portion, you can run just the leaf certificate portion:
Example 10:
/CA: - Version 1 - Deployments signs: - /Leaf 1 - Version 1 - Deployments
Single leaf version:
maestro regenerate leaf --name "/Leaf1"
CA signing leaf version:
maestro regenerate leaf --signed-by "/CA"
There should now be a new version of the leaf certificates.
Example 11:
/CA: - Version 1 - Deployments signs: - /Leaf 1 - Version 2 - Version 1 - Deployments
We need to now have these new leaf versions consumed by their deployments:
Apply Change on tiles listed in /Leaf version 1 deployments. If the deployment includes service instance tile make sure to utilize "upgrade" errand.
Example 12:
/CA: - Version 1 - Deployments signs: - /Leaf 1 - Version 2 - Deployments - Version 1
Feedback
