vRA Cloud Infoblox Plugin throws a certificate chain error during authentication process

Products

VMware Aria Suite

Issue/Introduction

This process fixes the symptom by resolving the root cause.

This is not a workaround.

This process fixes an improper configuration on the Infoblox server side.

Symptoms

When adding a new IPAM Infoblox integration to Cloud Assembly, when validating the integration, you may receive the following error message

Unable to validate the provided access credentials: Failed to validate credentials. AdapterReference: http://provisioningservice.prelude.svc.cluster.local:8282/provisioning/adapter/ipam/endpointconfig. Error: Execution of action Infoblox_ValidateEndpoint failed on provider side: Infoblox HTTP request failed with: HTTPSConnectionPool(host=’<FQDN>’, port=443): Max retries exceeded with url: /wapi/v2.7/networkview?_return_fields=name (Caused by SSLError(SSLError(“bad handshake: Error([(‘SSL routines’, ‘tls_process_server_certificate’, ‘certificate verify failed’)],)”,),)) Cloud account: null Task: /provisioning/endpoint-tasks/<endpoint_id> (less)

Environment

VMware Aria Automation 8.x

Cause

When establishing SSL handshake with the Infoblox server, Aria Automation relies on Infoblox to present a complete certificate chain - including server cert, intermediate and CA.

This is not a hard requirement for browsers since the HTTPS RFC dictates that servers are allowed to only present the server certificate and still a chain of trust can be built in case the intermediate and CA are stored in the browser certificate trust store.

However, Python 3.x is a more restrictive than browsers as it requires the full certificate chain in order to build the chain of trust. Since the Aria Automation Infoblox plugin is based on Python, customers must make sure that their Infoblox appliance is configured to return the whole certificate chain and not just the server certificate.

Resolution

There are 2 options to resolve this issue.

Option 1

Set the Infoblox.IPAM.DisableCertificateCheck parameter to True and Save the endpoint.

This will disable the SSL certificate checks so you won't get any more errors. However, from security perspective this is not the safest option since this opens the door for MITM attacks.

Option 2

Configure Infoblox to return the full certificate chain, including intermediate and CA.
This is the safest and recommended option.

Procedure

Verify that the Infoblox server only returns the server certificate, omitting the full certificate chain by running the following command
```
openssl s_client -showcerts -connect <hostname>:443
```

Verify the response is similar to the below exert

root@AriaAutoFQDN:~/openssl_test/root/ca$ openssl s_client -showcerts -connect <FQDN>:443
CONNECTED(00000003)
depth=0 C = BG, ST = XXXXX, L = XXXXX, O = XXXXXX, OU = XXXX, CN = <FQDN>, emailAddress = <Email_id>
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 C = BG, ST = XXXXX, L = XXXXX, O = XXXXXX, OU = XXXX, CN = <FQDN>, emailAddress = <Email_id>
verify error:num=21:unable to verify the first certificate
verify return:1

Note: Notice how the returned server certificate cannot be verified due to unable to verify the first certificate error.

Open a client browser and navigate to the Infoblox server domain. In the browser top right corner next to the URL there should be a button to view the certificate chain:
Click on the Certificate button and check the certificate path
The browser displays the full certificate chain - including intermediate and CA.
In case the browser does not display the intermediate certificate and the CA - contact the Infoblox server administrator and ask him to provide the complete chain of signer certificates that were used for signing the Infoblox server CSR.
Click on every certificate from the Certification Path tab except the server certificate and export it in PEM format:
Alice Ltd Intermediate CA > View Certificate > Details > Copy to File > Base 64 encoded X.509 (.CER) > Save
Alice Ltd Root CA > View Certificate > Details > Copy to File > Base 64 encoded X.509 (.CER) > Save
Concatenate the intermediate certificate and the CA into a single .pem file. The order in which the certificates are stored within the .pem file is very important. The CA must be at the last in the file, with each signer from the chain on top.
In our example the Intermediate CA must be first, followed by Root CA. It should look similar to
```
-----BEGIN CERTIFICATE-----
<INTERMEDIATE CA CERTIFICATE HERE>
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
<ROOT CA CERTIFICATE HERE>
-----END CERTIFICATE-----
```
Navigate to Infoblox > Grid > Members > Certificates > Manage CA Certificates

Upload the newly created .pem file from step 5. You should see the certificates in the popup.

Wait for 2-3 minutes until Infoblox picks up the changes and verify that the full certificate chain is now returned by running the following command

openssl s_client -showcerts -connect <hostname>:443

Example:

root@AriaAutoFQDN:~/openssl_test/root/ca$ openssl s_client -showcerts -connect <FQDN>:443
CONNECTED(00000003)
depth=2 C = GB, ST = XXXXXX, O = XXXXX Ltd, OU = XXXXX Ltd Certificate Authority, CN = XXXXX Ltd Root CA
verify error:num=19:self signed certificate in certificate chain
verify return:1
depth=2 C = GB, ST = XXXXXXX, O = XXXXX Ltd, OU = XXXXX Ltd Certificate Authority, CN = XXXXX Ltd Root CA
verify return:1
depth=1 C = GB, ST = XXXXXXX, O = XXXXX Ltd, OU = XXXXX Ltd Certificate Authority, CN = XXXXX Ltd Intermediate CA
verify return:1
depth=0 C = BG, ST = XXXXX, L = XXXXXXX, O = XXXXX, OU = XXXX, CN = <FQDN>
verify return:1

Note: As can be seen from the output, the Infoblox appliance now returns the full certificate chain.

Navigate to Aria Automation and change Infoblox.IPAM.DisableCertificateCheck to False. Click Validate.