Aria Operation cloud proxy shows offline in Aria Operation Product UI after successful cluster upgrade from 8.6.x to 8.10.x but CP version is still showing 8.6.2 in UI
search cancel

Aria Operation cloud proxy shows offline in Aria Operation Product UI after successful cluster upgrade from 8.6.x to 8.10.x but CP version is still showing 8.6.2 in UI

book

Article ID: 312288

calendar_today

Updated On:

Products

VMware Aria Suite

Issue/Introduction

  • Provide guidance or recommendations to address the "Cloud Proxy Offline" issue after vROPS upgrade from 8.6.x to 8.10.x versions.


Symptoms:

vROPS cloud proxy shows offline in Aria Operation Product UI after successful cluster upgrade from 8.6.x to 8.10.x but CP version is still showing 8.6.2 in UI and cprc-cli command shows CP version 8.10.2 and CP was offline and not collecting data.

 

Below events can be seen in Cloud Proxy logs in haproxy-traffic.log file:

2023-03-14T15:34:23+00:00 localhost haproxy[4014]: backend PrxyRC_BE has no server available!
2023-03-14T15:34:24+00:00 localhost haproxy[4014]: x.x.x.x:58764 [14/Mar/2023:15:34:21.489] PrxyRC_FE~ PrxyRC_BE/VROPS_4 0/0/-1/-1/3021 503 222 - - SC-- 4/4/0/0/3 0/0 "POST /suite-api/api/auth/token/acquire?_no_links=true HTTP/1.1"
2023-03-14T15:34:25+00:00 localhost haproxy[5451]: Proxy PrxyRC_FE started.
2023-03-14T15:34:25+00:00 localhost haproxy[5451]: Proxy PrxyRC_BE started.
2023-03-14T15:34:25+00:00 localhost haproxy[5451]: Proxy PrxyRC_UNSECURE_FE started.
2023-03-14T15:34:25+00:00 localhost haproxy[5451]: Proxy PrxyRC_UNSECURE_BE started.
2023-03-14T15:34:25+00:00 localhost haproxy[5454]: Server PrxyRC_BE/VROPS_0 is DOWN, reason: Layer6 invalid response, info: "SSL handshake failure", check duration: 10ms. 4 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2023-03-14T15:34:29+00:00 localhost haproxy[5454]: x.x.x.x:60922 [14/Mar/2023:15:34:26.651] PrxyRC_FE~ PrxyRC_BE/VROPS_1 0/0/-1/-1/3025 503 222 - - SC-- 2/2/1/0/3 0/0 "POST /casa/authorize HTTP/1.1"
2023-03-14T15:34:30+00:00 localhost haproxy[5454]: x.x.x.x:60934 [14/Mar/2023:15:34:27.005] PrxyRC_FE~ PrxyRC_BE/VROPS_2 0/0/-1/-1/3027 503 222 - - SC-- 2/2/0/0/3 0/0 "POST /suite-api/api/auth/token/acquire?_no_links=true HTTP/1.1"
2023-03-14T15:34:35+00:00 localhost haproxy[5454]: x.x.x.x:60922 [14/Mar/2023:15:34:32.710] PrxyRC_FE~ PrxyRC_BE/VROPS_3 0/0/-1/-1/3023 503 222 - - SC-- 3/3/1/0/3 0/0 "POST /casa/authorize HTTP/1.1"
2023-03-14T15:34:35+00:00 localhost haproxy[5454]: x.x.x.x:60944 [14/Mar/2023:15:34:32.741] PrxyRC_FE~ PrxyRC_BE/VROPS_4 0/0/-1/-1/3028 503 222 - - SC-- 3/3/0/0/3 0/0 "POST /casa/authorize HTTP/1.1"
2023-03-14T15:34:40+00:00 localhost haproxy[5454]: Server PrxyRC_BE/VROPS_1 is DOWN, reason: Layer6 invalid response, info: "SSL handshake failure", check duration: 4ms. 3 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2023-03-14T15:34:48+00:00 localhost haproxy[5454]: x.x.x.x:60922
[14/Mar/2023:15:34:45.773] PrxyRC_FE~ PrxyRC_BE/VROPS_2 0/0/-1/-1/3025 503 222 - - SC-- 3/3/0/0/3 0/0 "POST /casa/authorize HTTP/1.1"
2023-03-14T15:34:55+00:00 localhost haproxy[5454]: Server PrxyRC_BE/VROPS_2 is DOWN, reason: Layer6 invalid response, info: "SSL handshake failure", check duration: 5ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2023-03-14T15:35:01+00:00 localhost haproxy[5454]: x.x.x.x:60944
[14/Mar/2023:15:34:58.825] PrxyRC_FE~ PrxyRC_BE/VROPS_3 0/0/-1/-1/3023 503 222 - - SC-- 3/3/0/0/3 0/0 "POST /casa/authorize HTTP/1.1"
2023-03-14T15:35:10+00:00 localhost haproxy[5454]: Server PrxyRC_BE/VROPS_3 is DOWN, reason: Layer6 invalid response, info: "SSL handshake failure", check duration: 5ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2023-03-14T15:35:14+00:00 localhost haproxy[5454]: x.x.x.x:60922
 [14/Mar/2023:15:35:11.875] PrxyRC_FE~ PrxyRC_BE/VROPS_4 0/0/-1/-1/3025 503 222 - - SC-- 3/3/0/0/3 0/0 "POST /casa/authorize HTTP/1.1"
2023-03-14T15:35:25+00:00 localhost haproxy[5454]: Server PrxyRC_BE/VROPS_4 is DOWN, reason: Layer6 invalid response, info: "SSL handshake failure", check duration: 5ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2023-03-14T15:35:25+00:00 localhost haproxy[5454]: backend PrxyRC_BE has no server available!

 


Environment

VMware vRealize Operations 8.x

Cause

  • The reason is that customer applied a custom certificate on the vROps cluster. As the version was 8.6, Certificates were not automatically transferred to CP (it is in place starting from 8.10), so after the cluster upgrade CP was not able to do a certificate renewal, which caused that connections from CP to vROps cluster to fail due to untrusted certificates.

Resolution

  • This issue will not occur in Aria Operations (vROPS) version 8.10.x


Workaround:
Note: Please take Snapshot of the Cloud Proxy(s) before implementing the Workaround.
 
  • Below steps were implemented to Resolve the issue for Custom vROPS SSL Certs:

 

  1. Open SSH sessions on vROPS Primary Node and Cloud Proxy Node and stop below services on CP:
    1. service vmware-casa stop
    2. service httpd-north stop
    3. service haproxy stop
    4. service collector stop
  2. Copy contents of web_chain.pem and cacert.pem files in cd /storage/vcops/user/conf/ssl (in vROPS Master) using cat /storage/vcops/user/conf/ssl/web_chain.pem and /storage/vcops/user/conf/ssl/cacert.pem
  3. Backup server.ca.pem file in cd /storage/vcops/user/conf/ssl (in Cloud Proxy) using "cp server.ca.pem server.ca.pem_bak" and create a file named server.root.ca.pem using "cp server.ca.pem server.root.ca.pem" command in cd /storage/vcops/user/conf/ssl (in Cloud Proxy) and paste the contents of web_chain.pem file in server.ca.pem file and contents of cacert.pem in server.root.ca.pem file in CP.
  4. At this point vROPS Cluster's web_chain.pem content should have been copied to server.ca.pem in Cloud Proxy and content of vROPS Cluster's cacert.pem should have been copied to server.root.ca.pem in Cloud Proxy.
  5. Assign appropriate file ermission using below command:
    1. chmod 644 server.ca.pem
    2. chmod 644 server.root.ca.pem
  6. Then assign file ownership using below commands:
    1. chown admin:admin server.ca.pem
    2. chown admin:admin server.root.ca.pem
  7. Reboot Cloud Proxy using reboot -f command


Additional Information

Impact/Risks:
  • CP will show in Offline state in vROPS Product UI and CP version will be 8.6.x. However, using cprc-cli -s command, CP version will report as 8.10.x
  • CP collection will stop.


Powered by Wolken Software