Cloud Proxy shows as Offline after upgrading VMware Aria Operations

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• After upgrading the Cloud Proxy, it's status shows as Offline in the Product UI.
  
  
• Cloud Proxy VMs are stuck on Upgrading status in Aria Operations Admin UI  even though upgrade is completed.

• The /var/log/haproxy-admin.log on the Cloud Proxy shows errors similar to:

Server PrxyRC_BE/VROPS_0 is DOWN, reason: Layer6 invalid response, info: "SSL handshake failure", check duration: 161ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.

Environment

Aria Operations 8.17.x

Aria Operations 8.18.x

Cause

This issue occurs when SSL inspection is enabled on the local network (firewall or proxy) and the certificate chain used for inspection is not present on the Cloud Proxy. It can also occur if the Aria Operations certificate has expired or changed during the upgrade process, leading to a trust failure.

Resolution

To resolve this issue, Recommended Fix is to follow the Add CA certs to a cloud proxy in VMware Aria Operations

Workaround:

To workaround the issue, the Cloud Proxy should be whitelisted in the network firewall configuration.

Alternatively, you can set ssl verify to none on the Cloud Proxy.

To do this, complete the following steps:

1. 1. Log into the Cloud Proxy as root via SSH or Console.

   2. Run the following command to backup the /etc/haproxy/haproxy.cfg file:

      cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.bak

   3. Open /etc/haproxy/haproxy.cfg in a text editor.

   4. Change all occurrences of ssl verify required ca-file /storage/vcops/user/conf/ssl/haproxy.ca.pem to ssl verify none.

   5. Save and close the file.

   6. Run the following command to restart the haproxy and collector services:

service haproxy restart && service collector restart

Note: The above changes are not persistent across CP node reboot or upgrades.

Additional Information

To undo the workaround of setting ssl verify to none, complete the following:

1. Log into the Cloud Proxy as root via SSH or Console.
2. Run the following command to replace the /etc/haproxy/haproxy.cfg file with the backed up version:

mv /etc/haproxy/haproxy.bak /etc/haproxy/haproxy.cfg

3. Run the following command to restart the haproxy and collector services:

service haproxy restart && service collector restart