Cloud Proxy httpd-south Service Fails Due to Missing Certificates Post-Upgrade
Article ID: 435912
Updated On:
Products
Issue/Introduction
After upgrading the Cloud Proxy appliance from version 8.18.3 to 9.0.x, cloud proxy operations are impacted, and the httpd-south.service fails to start. This failure occurs because a new cloud-proxy server certificate cannot be generated.
Reviewing the /var/log/httpd-south/error_log reveals the following exact error message indicating an SSL certificate mismatch:
[ssl:emerg] [pid 25461:tid 25461] AH02565: Certificate and private key <CloudProxyIP>:8443:0 from /ucp/ssl/root/ca/certs/ucpapi.cert.pem and /ucp/ssl/root/ca/private/ucpapi.key do not match
[ssl:emerg] [pid 25791:tid 25791] AH02565: Certificate and private key <CloudProxyIP>:8443:0 from /ucp/ssl/root/ca/certs/ucpapi.cert.pem and /ucp/ssl/root/ca/private/ucpapi.key do not match
Environment
Aria Operations Cloud Proxy 8.18.3
VCF Operations Cloud Proxy 9.0.x
Cause
During the upgrade process to 9.0.x, the internal CA certificate and its corresponding key pair are unexpectedly deleted. Without the CA certificate and key, the Cloud Proxy cannot generate the required server certificate, resulting in the httpd-south.service failing to start.
Resolution
To resolve this issue, manually recreate the CA certificate and server certificate, then restart the impacted service.
1. Connect to the affected Cloud Proxy appliance via SSH.
2. Navigate to the internal CA directory:
cd /ucp/ssl/root/ca
3. Generate the new CA certificate and its associated keys:
openssl genrsa -out private/ca.key 2048 openssl pkcs8 -topk8 -v2 aes256 -passout pass:Ucp~123 -inform pem -in private/ca.key -outform pem -out private/ca.key.pem openssl req -passin pass:Ucp~123 -subj "/C=IN/ST=KA/L=BLR/O=Broadcom, Inc./OU=Broadcom, Inc./CN=ucpca" -config openssl.cnf -key private/ca.key.pem -new -x509 -days 7300 -sha256 -extensions v3_ca -out certs/ca.cert.pem
4. Generate the new server certificate keys:
openssl genrsa -out /ucp/ssl/root/ca/private/ucpapi.key 2048 openssl rsa -in /ucp/ssl/root/ca/private/ucpapi.key -pubout > /ucp/ssl/root/ca/private/ucpapi.pub openssl pkcs8 -topk8 -v2 aes256 -passout pass:Ucp~123 -inform pem -in /ucp/ssl/root/ca/private/ucpapi.key -outform pem -out /ucp/ssl/root/ca/private/ucpapi.key.pem
5. Inspect the /ucp/config/config.properties file and verify the value of UCP_FQDN_4AGENT_CONFIGURED.
6. If UCP_FQDN_4AGENT_CONFIGURED is true:
a. Identify the UCP_FQDN_4AGENT value in /ucp/config/config.properties and replace it in the next command
b. openssl req -passin pass:Ucp~123 -subj "/C=IN/ST=KA/L=BLR/O=Broadcom, Inc./OU=Broadcom, Inc./CN=UCP_FQDN_4AGENT/[email protected]" -config /ucp/ssl/root/ca/openssl.cnf -addext "subjectAltName = DNS:UCP_FQDN_4AGENT" -key /ucp/ssl/root/ca/private/ucpapi.key.pem -newkey rsa:2048 -nodes -sha256 -out /ucp/ssl/root/ca/csr/ucpapi.csr.pem -days 7300
7. If UCP_FQDN_4AGENT_CONFIGURED is false:
a. Identify the UCP_FQDN_4AGENT value in /ucp/config/config.properties and replace it in the next command
b. openssl req -passin pass:Ucp~123 -subj "/C=IN/ST=KA/L=BLR/O=Broadcom, Inc./OU=Broadcom, Inc./CN=ucpapi" -config /ucp/ssl/root/ca/openssl.cnf -addext "subjectAltName = IP:UCP_FQDN_4AGENT" -key /ucp/ssl/root/ca/private/ucpapi.key.pem -newkey rsa:2048 -nodes -sha256 -out /ucp/ssl/root/ca/csr/ucpapi.csr.pem -days 7300
8. Generate the final server certificate:
openssl ca -batch -passin pass:Ucp~123 -config /ucp/ssl/root/ca/openssl.cnf -extensions server_cert -days 7300 -notext -md sha256 -in /ucp/ssl/root/ca/csr/ucpapi.csr.pem -out /ucp/ssl/root/ca/certs/ucpapi.cert.pem
9. Restart the httpd-south.service to apply the newly generated certificates:
systemctl restart httpd-south.service
Additional Information
If there are issues with VMs that are unable to Send Metrics to Cloud Proxy via Telegraf in Brownfield Deployments, please refer to this KB
Feedback
