RabbitMQ Smoke tests fail after enabling TLS for on-demand service

Products

VMware RabbitMQ

Issue/Introduction

Symptoms:
After enabling TLS for RabbitMQ Tile On-Demand service, the smoke tests fail to update the instance during the step to enable TLS in Rabbit.

The error message from Operations (Ops) Manager is as follows:

status: update failed
message: Instance update failed: There was a problem completing your request. Please contact your operations team providing the following information: service: p.rabbitmq, service-instance-guid: cf52f413-992d-4361-8ac5-c4923e4cbb11, broker-request-id: 1e9b938e-4226-4384-9b04-50868d84361d, task-id: 2915, operation: update

There were no errors observed during the steps to prepare for TLS. Checking the task in question, 2915 in this example, the error is as follows:

Task 2915

Task 2915 | 10:55:18 | Preparing deployment: Preparing deployment (00:00:01)
                     L Error: Config Server failed to generate value for '/p-bosh/service-instance_cf52f413-992d-4361-8ac5-c4923e4cbb11/tls_certificate' with type 'certificate'. HTTP Code '400', Error: 'The requested certificate cannot be signed by the given CA because the given CA is not a certificate authority. A certificate must contain the extension 'Certificate Authority: YES' to be used to sign other certificates.'
Task 2915 | 10:55:20 | Error: Config Server failed to generate value for '/p-bosh/service-instance_cf52f413-992d-4361-8ac5-c4923e4cbb11/tls_certificate' with type 'certificate'. HTTP Code '400', Error: 'The requested certificate cannot be signed by the given CA because the given CA is not a certificate authority. A certificate must contain the extension 'Certificate Authority: YES' to be used to sign other certificates.'

Task 2915 Started  Mon Sep 10 10:55:18 UTC 2018
Task 2915 Finished Mon Sep 10 10:55:20 UTC 2018
Task 2915 Duration 00:00:02
Task 2915 error

Capturing task '2915' output:
  Expected task '2915' to succeed but state is 'error'

Exit code 1

Environment

Cause

The error in the task The requested certificate cannot be signed by the given CA because the given CA is not a certificate authority reports that the certificate used for RabbitMQ TLS is not a Certificate Authority (CA), therefore it is not allowed to sign other certificates.

We can verify this information running the following commands:

# 1. Login into CredHub
credhub login --client-name=<client-name> --client-secret=<client-secret>
# 2. Obtain the public certificate 
bosh2 int <(credhub get --name=/services/tls_ca) --path /value/certificate
# 3. Copy the output into a file e.g. mycert.pem
# 4. Run the following SSL command
openssl x509 -noout -text -in mycert.pem

The client name and secret for command (1) was set during step 7 of "Preparing for TLS" [1]. The Open SSL command (4) does not output any private or secret information.

The Open SSL command (4) output should contain the following information:

# openssl x509 -noout -text -in mycert.pem
Certificate:
    Data:
        Version: 3 (0x2)
[...]
X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
[...]
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
[...]

The basic constraint CA: False makes this certificate a non-CA certificate, therefore it is not able to sign other certificates.

[1] https://docs.vmware.com/en/VMware-Tanzu-RabbitMQ/index.html

Resolution

Use a Certificate Authority certificate. You will need access to the private key associated to this certificate and repeat the step 'Add the CA Certificate' [2]
Generate your own CA certificate using CredHub credhub generate command

[2] https://docs.vmware.com/en/VMware-Tanzu-RabbitMQ/index.html