Out of sync CA causes Maestro safety violation

Products

• Operations Manager

Issue/Introduction

During a certificate rotation procedure featuring Ops Manager 2.10, it is possible to get a safety violation which states that "there is more than one signing version of a certificate authority".

Environment

Product Version: 2.10

Resolution

Version Requirements:

This issue requires a certain level of Maestro to troubleshoot, with the version on the OpsMan VM being the preferable one (2.10 uses maestro 8). Earlier versions will not display "signed_by_version" in the output of "topology". 
 

CA Example:

We can see the issue in the following topology output of our test ca. We have trimmed it to show the relevant fields. Make note of the duplicate "signing: true" fields:

maestro topology --name="/test_ca"
    - name: /test_ca
      certificate_id: 00000000-0000-0000-0000-000000000000
      signed_by: /test_ca
      versions:
    A)    - version_id: 11111111-1111-1111-1111-111111111111  
          signing: true       <--- Both are signing
          certificate_authority: true
          valid_until: 2022-02-22T09:52:01Z      <--- The New Certificate
    B)    - version_id: 22222222-2222-2222-2222-222222222222  
          signing: true       <--- Both are signing
          certificate_authority: true
          valid_until: 2021-03-13T13:19:41Z      <--- The Old Certificate
      signs:
    C)    - name: /cert_charlie
              signed_by_version: 22222222-2222-2222-2222-222222222222
              deployment_names:
                - /deployment_charlie
    D)    - name: /cert_delta
              signed_by_version: 11111111-1111-1111-1111-111111111111
              deployment_names:
                - /deployment_delta
    E)    - name: /cert_echo
              signed_by_version: 11111111-1111-1111-1111-111111111111
              deployment_names:
                - deployment_echo
    F)    - name: /cert_foxtrot
              signed_by_version: 11111111-1111-1111-1111-111111111111
              deployment_names:
                - deployment_foxtrot
    G)    - name: /cert_golf
              signed_by_version: 11111111-1111-1111-1111-111111111111
              deployment_names:
                - deployment_golf
    H)    - name: /cert_hotel
              signed_by_version: 22222222-2222-2222-2222-222222222222
              deployment_names:
                - deployment_hotel
                - deployment_quebec

In this circumstance we can see that there are 2 leaf certificates (C & H) being signed by the signing CA B. We need to force those 2 leaf certificates to "regenerate" off of signing CA A. The command "maestro regenerate leaf --name="/cert_hotel" will error as well given that there are two CAs available. 

The resolution is to run the following:

maestro regenerate leaf --name="/cert_hotel" --skip-safety-check

By doing this we force it to regenerate off of the defaulting CA, the newest CA (A). You repeat this for each certificate being signed off of CA B, in this case C and H. 

 

An example of the action plan

Utilizing maestro to perform the regeneration on the afflicted leaf certificates:

maestro regenerate leaf --name="/cert_charlie" --skip-safety-check
maestro regenerate leaf --name="/cert_hotel" --skip-safety-check

Perform an Apply Change with "upgrade" errands on the tiles that correspond with their deployments:

•  deployment_charlie
•  deployment_hotel
•  deployment_quebec

Once complete, resume certificate rotation step where the error was thrown. 
 

For multiple CAs

Repeat troubleshooting process for every afflicted CA.