Aria Operations node stuck on "Waiting for Analytics" status

Products

VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

Cluster cannot be brought online because cluster node is not Online.
The node is stuck in "Waiting for Analytics" state.
Analytics service is not running.

Environment

Aria Operations 8.18.x

Cause

The vPostgres certificate was manually updated with the custom certificate by modifying the following files:

/storage/vcops/user/conf/ssl/cacert.pem
/var/vmware/vpostgres/current/vpostgres_cacert.pem
/var/vmware/vpostgres/current/vpostgres_key.pem

The following log exception is observed in the analytics log file:

[Timestamp]+0000 WARN  [C3P0PooledConnectionPoolManager[identityToken->################|######]-HelperThread-#6]  com.mchange.v2.resourcepool.BasicResourcePool.log - com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@##### -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception:
org.postgresql.util.PSQLException: The SSLSocketFactory class provided org.postgresql.ssl.LibPQvROpsFactory could not be instantiated.
        at org.postgresql.Driver$ConnectThread.getResult(Driver.java:385) ~[postgresql-42.5.1.jar:42.5.1]
        at org.postgresql.Driver.connect(Driver.java:298) ~[postgresql-42.5.1.jar:42.5.1]
Caused by: org.postgresql.util.PSQLException: Could not open SSL root certificate file /storage/vcops/user/conf/ssl/cacert.pem.
        at org.postgresql.ssl.LibPQvROpsFactory.<init>(LibPQvROpsFactory.java:153) ~[vcops-postgres-sslfactory-1.0-SNAPSHOT.jar:42.5.1]
Caused by: java.io.FileNotFoundException: /storage/vcops/user/conf/ssl/cacert.pem (Permission denied)
        at java.io.FileInputStream.open0(Native Method) ~[?:?]
        at java.io.FileInputStream.open(Unknown Source) ~[?:?]

Resolution

NOTE: Modification of the Internal vPostgres certificate is not advised as it will potentially break the internal database communication, causing analytics service to stop running.

Perform the following steps to revert the changes.

Take Snapshot of all cluster nodes prior to making any changes.
SSH into the Primary node as 'root'.

Run the following command to bring the node offline.

$VMWARE_PYTHON_BIN $VCOPS_BASE/../vmware-vcopssuite/utilities/sliceConfiguration/bin/vcopsConfigureRoles.py --action bringSliceOffline

Revert the updated internal certificate files by issuing the following commands:

cd /storage/vcops/user/conf/ssl/
cp -p cacert.pem.old cacert.pem
cd /var/vmware/vpostgres/current/
cp -p vpostgres_cacert.pem.old vpostgres_cacert.pem
cp -p vpostgres_key.pem.old vpostgres_key.pem

Bring node back online by running the below command:

$VMWARE_PYTHON_BIN $VCOPS_BASE/../vmware-vcopssuite/utilities/sliceConfiguration/bin/vcopsConfigureRoles.py --action bringSliceOnline

Bring cluster online by running the below command

$VMWARE_PYTHON_BIN /usr/lib/vmware-vcopssuite/utilities/sliceConfiguration/bin/vcopsClusterManager.py init-cluster

Wait for 10-15 minutes to confirm that the cluster is going online.

If the above steps does not resolve the issue, raise a 'Broadcom Support Request' referring to this KB for further assistance.