SSO UI fails to load in VCF Operations 9.x due to Cloud Proxy Offline status

Products

• VCF Operations
• VMware Cloud Foundation

Issue/Introduction

• When navigating to Fleet Management > Identity & Access in VCF Operations, the SSO configuration page fails to load with the following error in the screenshot
• Attempts to enable SSO for a VCF instance result in the error:
  "There must be at least one licensed VCF instance with a management domain vCenter to enable SSO."
• The Cloud Proxy or Collection Appliance associated with the Management Domain may show a status of Warning or Offline in the VCF Operations UI.
• The Cloud Proxy Health Status may report an "Outdated life status."
  
  VCF Operations > VCF Fleet Management > Identity & Access
  
  Or
  
  VCF Operations > Administrator > SDDC Manager
  

Environment

VCF 9.x

Cause

Management services (HAProxy) on the Cloud Proxy become unresponsive due to:

• DNS Resolution Failures: HAProxy requires successful FQDN resolution for all nodes to start.
• SSL Trust Mismatches: Discrepancies between certificates on the Cloud Proxy and the Primary node.
• Infrastructure Changes: Moving VMs between network segments (e.g., NSX overlay to local networking) can invalidate DNS settings

Resolution

To resolve this issue, follow the steps below:

Step 1: Standard Service Recovery

1. Verify DNS: SSH into the Cloud Proxy and ensure it can resolve the FQDN of the VCF Operations nodes using nslookup.
   Note: Verification must include all cluster nodes (Primary and all Data nodes)
   
   
2. Restart Services on Cloud Proxy appliance: SSH to the appliance as Root
   
   systemctl restart haproxy
service collector restart
   
    
3. Appliance Reboot: If services do not recover, perform a graceful reboot of the Cloud Proxy appliance.

Note: If Step 1 does not resolve the issue, please follow Step 2.

Step 2: Addressing SSL Handshake Failures (Workaround from KB 331408) 

If the proxy remains offline due to SSL handshake errors and cannot be immediately whitelisted in the firewall, apply the following workaround to disable SSL verification:

1. Backup Configuration:
   
   
   cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.bak

    
2. Modify HAProxy Config: vi /etc/haproxy/haproxy.cfg in a text editor
   
   
3. Disable SSL Verify:
   
   Change all occurrences of
   
   From: ssl verify required ca-file /storage/vcops/user/conf/ssl/haproxy.ca.pem 
   
   To:       ssl verify none
   
   
4. Restart service to apply changes:
   
   
   service haproxy restart && service collector restart

    
   Note: These changes are not persistent across appliance reboots or upgrades. The permanent fix is to ensure the correct CA certificate chain is present on the proxy.

Step 3: Certificate Chain Comparison. If the proxy remains offline, verify the certificate chain matches the Primary node:

Verification (Comparing the Chains)

You need to compare the certificates.

On the Primary Node

Run these commands to display the contents of the active certificate chain:

# View the Web Chain (End-entity + Intermediates)
cat /storage/vcops/user/conf/ssl/web_chain.pem

# View the Trusted CA certificates
cat /storage/vcops/user/conf/ssl/cacert.pem

On the Cloud Proxy Node

Run these commands and compare the output to the Primary node:

# Compare this against the web_chain.pem from Primary
cat /storage/vcops/user/conf/ssl/server.ca.pem

# Compare this against the cacert.pem from Primary
cat /storage/vcops/user/conf/ssl/server.root.ca.pem

Resolution: Updating the Proxy if the files do not match, follow these steps to manually sync them.

1. Backup existing files on the Cloud Proxy:
   
   cp /storage/vcops/user/conf/ssl/server.ca.pem /storage/vcops/user/conf/ssl/server.ca.pem.bak
cp /storage/vcops/user/conf/ssl/server.root.ca.pem /storage/vcops/user/conf/ssl/server.root.ca.pem.bak

2. Update the files: You can use vi or nano to paste the content from the Primary node into the Proxy files, or use scp to copy them directly from the Primary node:
   
   # Run this on the Cloud Proxy to pull from Primary
   scp root@[Primary_Node_IP]:/storage/vcops/user/conf/ssl/web_chain.pem /storage/vcops/user/conf/ssl/server.ca.pem
scp root@[Primary_Node_IP]:/storage/vcops/user/conf/ssl/cacert.pem /storage/vcops/user/conf/ssl/server.root.ca.pem

3. Correct Permissions: Ensure the files have the correct ownership:
   
   chown admin:admin /storage/vcops/user/conf/ssl/server.ca.pem
chown admin:admin /storage/vcops/user/conf/ssl/server.root.ca.pem

4. Restart Services: Restart the Casa service to pick up the new certificates:
   
   service vmware-casa restart