Out of sync CA causes Maestro safety violation

Products

• Operations Manager

Issue/Introduction

During a certificate rotation procedure featuring Ops Manager 3.x, it is possible to get a safety violation which states that "there is more than one signing version of a certificate authority".

Environment

Ops Manager 3.x

Cause

This occurs when the certificate rotation procedure is interrupted (e.g., a stopped pipeline), leaving both old and new versions of a Certificate Authority (CA) marked with signing: true.

Resolution

1. Identify the out-of-sync CA using the Maestro topology:
   

   maestro topology --name="/example_ca"

    

2. Look for two versions where signing: true. Identify which leaf certificates are still signed by the older version (e.g., /cert_hotel signed by the old version ID).

   See the issue in the following topology output of our test ca, which has been trimmed to show the relevant fields. 
   

   maestro topology --name="/test_ca"
       - name: /test_ca
         certificate_id: 00000000-0000-0000-0000-000000000000
         signed_by: /test_ca
         versions:
       A)    - version_id: 11111111-1111-1111-1111-111111111111  
             signing: true       <--- Both are signing
             certificate_authority: true
             valid_until: 2022-02-22T09:52:01Z      <--- The New Certificate
       B)    - version_id: 22222222-2222-2222-2222-222222222222  
             signing: true       <--- Both are signing
             certificate_authority: true
             valid_until: 2021-03-13T13:19:41Z      <--- The Old Certificate
         signs:
       C)    - name: /cert_charlie
                 signed_by_version: 22222222-2222-2222-2222-222222222222
                 deployment_names:
                   - /deployment_charlie
       D)    - name: /cert_delta
                 signed_by_version: 11111111-1111-1111-1111-111111111111
                 deployment_names:
                   - /deployment_delta
       E)    - name: /cert_echo
                 signed_by_version: 11111111-1111-1111-1111-111111111111
                 deployment_names:
                   - deployment_echo
       F)    - name: /cert_foxtrot
                 signed_by_version: 11111111-1111-1111-1111-111111111111
                 deployment_names:
                   - deployment_foxtrot
       G)    - name: /cert_golf
                 signed_by_version: 11111111-1111-1111-1111-111111111111
                 deployment_names:
                   - deployment_golf
       H)    - name: /cert_hotel
                 signed_by_version: 22222222-2222-2222-2222-222222222222
                 deployment_names:
                   - deployment_hotel
                   - deployment_quebec

    

3. See that there are 2 leaf certificates (C & H) being signed by the signing CA B. You need to force those 2 leaf certificates to "regenerate" off of signing CA A. The command "maestro regenerate leaf --name="/cert_hotel" will error as well given that there are two CAs available. 
4. Force the afflicted leaf certificates to regenerate using the new CA:
   

   maestro regenerate leaf --name="/cert_hotel" --skip-safety-check

   This will force it to regenerate off of the defaulting CA, the newest CA (A). Repeat this for each certificate being signed off of CA B, in this case C and H. 

5. Perform an Apply Changes with "upgrade" errands on the affected tiles.
6. Once complete, resume certificate rotation step where the error was thrown. 

 

An example of the action plan

Utilizing maestro to perform the regeneration on the afflicted leaf certificates:

maestro regenerate leaf --name="/cert_charlie" --skip-safety-check
maestro regenerate leaf --name="/cert_hotel" --skip-safety-check

Perform an Apply Change with "upgrade" errands on the tiles that correspond with their deployments:

•  deployment_charlie
•  deployment_hotel
•  deployment_quebec

Once complete, resume certificate rotation step where the error was thrown. 
 

For multiple CAs

Repeat troubleshooting process for every afflicted CA.