Cloud Proxy httpd-south Service Fails Due to Missing Certificates Post-Upgrade

Products

• VCF Operations

Issue/Introduction

After upgrading the Cloud Proxy appliance from version 8.18.3 to 9.0.x, Cloud Proxy operations may become impacted due to the failure of the httpd-south.service to start. This issue occurs when the system is unable to generate a new Cloud-Proxy server certificate.

Upon reviewing the /var/log/httpd-south/error_log, the following error message can be observed, indicating an SSL certificate mismatch:

[ssl:emerg] [pid 25461:tid 25461] AH02565: Certificate and private key <CloudProxyIP>:8443:0 from /ucp/ssl/root/ca/certs/ucpapi.cert.pem and /ucp/ssl/root/ca/private/ucpapi.key do not match
[ssl:emerg] [pid 25791:tid 25791] AH02565: Certificate and private key <CloudProxyIP>:8443:0 from /ucp/ssl/root/ca/certs/ucpapi.cert.pem and /ucp/ssl/root/ca/private/ucpapi.key do not match

Environment

Aria Operations Cloud Proxy 8.18.3

VCF Operations Cloud Proxy 9.0.x

Cause

During the upgrade to version 9.0.x, the internal CA certificate and its associated key pair may be inadvertently removed. In the absence of these components, the Cloud Proxy is unable to generate the required server certificate, which leads to the failure of the httpd-south.service to start.

Resolution

To resolve this issue, manually regenerate the internal CA certificate along with the required server certificate, and then restart the affected service to restore normal operation.

• Connect to the affected Cloud Proxy appliance via SSH.
  
  
• Navigate to the internal CA directory:
  cd /ucp/ssl/root/ca
  
  
• Generate the new CA certificate and its associated keys:
  openssl genrsa -out private/ca.key 2048
openssl pkcs8 -topk8 -v2 aes256 -passout pass:Ucp~123 -inform pem -in private/ca.key -outform pem -out private/ca.key.pem
openssl req -passin pass:Ucp~123 -subj "/C=IN/ST=KA/L=BLR/O=Broadcom, Inc./OU=Broadcom, Inc./CN=ucpca" -config openssl.cnf -key private/ca.key.pem -new -x509 -days 7300 -sha256 -extensions v3_ca -out certs/ca.cert.pem
  
  
• Generate the new server certificate keys:
  openssl genrsa -out /ucp/ssl/root/ca/private/ucpapi.key 2048
openssl rsa -in /ucp/ssl/root/ca/private/ucpapi.key -pubout > /ucp/ssl/root/ca/private/ucpapi.pub
openssl pkcs8  -topk8 -v2 aes256 -passout pass:Ucp~123 -inform pem -in /ucp/ssl/root/ca/private/ucpapi.key -outform pem -out /ucp/ssl/root/ca/private/ucpapi.key.pem
  
  
• Inspect the /ucp/config/config.properties file and verify the value of UCP_FQDN_4AGENT_CONFIGURED.
  
  
• If UCP_FQDN_4AGENT_CONFIGURED is true:
  a. Identify the UCP_FQDN_4AGENT value in /ucp/config/config.properties and replace it in the next command
  b. openssl req  -passin pass:Ucp~123 -subj "/C=IN/ST=KA/L=BLR/O=Broadcom, Inc./OU=Broadcom, Inc./CN=UCP_FQDN_4AGENT/[email protected]" -config /ucp/ssl/root/ca/openssl.cnf -addext "subjectAltName = DNS:UCP_FQDN_4AGENT" -key /ucp/ssl/root/ca/private/ucpapi.key.pem -newkey rsa:2048 -nodes -sha256 -out /ucp/ssl/root/ca/csr/ucpapi.csr.pem -days 7300
  
  
• If UCP_FQDN_4AGENT_CONFIGURED is false:
  a. Identify the UCP_FQDN_4AGENT value in /ucp/config/config.properties and replace it in the next command
  b. openssl req  -passin pass:Ucp~123 -subj "/C=IN/ST=KA/L=BLR/O=Broadcom, Inc./OU=Broadcom, Inc./CN=ucpapi" -config /ucp/ssl/root/ca/openssl.cnf -addext "subjectAltName = IP:UCP_FQDN_4AGENT" -key /ucp/ssl/root/ca/private/ucpapi.key.pem -newkey rsa:2048 -nodes -sha256 -out /ucp/ssl/root/ca/csr/ucpapi.csr.pem -days 7300
  
  
• Generate the final server certificate:
  openssl ca -batch -passin pass:Ucp~123 -config /ucp/ssl/root/ca/openssl.cnf -extensions server_cert  -days 7300 -notext -md sha256 -in /ucp/ssl/root/ca/csr/ucpapi.csr.pem -out /ucp/ssl/root/ca/certs/ucpapi.cert.pem
  
  
• Restart the httpd-south.service to apply the newly generated certificates:
  systemctl restart httpd-south.service

Additional Information

If there are issues with VMs that are unable to Send Metrics to Cloud Proxy via Telegraf in Brownfield Deployments, please refer to this KB