Ops Manager Apply Changes operation is blocked and fails with "unable to get local issuer certificate errors"

Products

• Operations Manager

Issue/Introduction

Pre-requisites

• IaaS is vSphere
• ssl_verification_enabled flag is set to true. This is also verified in the resolution section of this article.

Description

Running Apply Changes operation from the Ops Manager UI or using the Platform Automation pipeline presents the following errors:

{
  "errors": [
    "IaaS default: 'SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)', type: PrivilegeVerifier",
    "IaaS default: 'SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)', type: IaasConfigurationVerifier",
    "IaaS default: 'SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)', type: AvailabilityZonesVerifier",
    "'SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)', type: NetworksExistenceVerifier"
  ]
....
....
....
}

On the UI, the error is shown as follows:

Environment

Product Version: 2.7

Resolution

Note: This article assumes that you have a single iaas_configuration (Single vCenter configured in Ops Manager UI -> Bosh Director tile -> vCenter Config).

1. Verify status of ssl_verification_enabled flag for your iaas_configuration by navigating to the following endpoint in a web browser:

https://<OPSMAN-URL>/api/v0/staged/director/iaas_configurations

2. From the output of the command above, if the value of ssl_verification_enabled flag is set to true, then it means the Ops Manager VM wants to communicate to vCenter using SSL.

3. The following steps can be followed to disable SSL verification to get past this issue:

Disabling SSL verification

• This can be done by updating the ssl_verification_enabled flag to false by querying the Ops Manager API endpoint that updates the IaaS configuration, /api/v0/staged/director/iaas_configurations/:guid.
• The detailed steps on how to update the iaas_configuration can be found in the Ops Manager API doc. 
• High-level steps can be summarized as follows:
  • Get the access token from Ops Manager UAA.
  • Run PUT query against Ops Manager's API endpoint (/api/v0/staged/director/iaas_configurations/:guid) by providing the full data (not just ssl_verification_enabled configuration) for the iaas_configuration JSON object for a vSphere installation.
  • An example of a PUT request for a vSphere installation can be found here.

Note: Keeping the ssl_verifiaction_enable flag to true and updating Ops Manager's VM (Linux kernel) to have vCenter CA in its truststore is out of scope for this article. This is currently a feature yet to be implemented in the Ops Manager's UI elements.