Aria Operations Data Node stuck in "Configuring" state with a message as "Preparing node for first use".

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• When adding a new Data Node to an existing Aria Operations (formerly vRealize Operations) cluster, the deployment hangs indefinitely in Admin UI at the "Configuring" stage with a message as "Preparing node for first use".
• Below log entries are observed in /storage/log/vcops/log/vcopsConfigureRoles.log on the new Data Node.

yyyy-mm-ddThh:mm:ss ERROR [6446] - root - vc_ops_utilities - logError - An unhandled exception occurred, exiting with exit code: 1,
Type: "<class 'urllib.error.URLError'>"
 Value: "<urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1006)>"
 Traceback: "Traceback (most recent call last):
  File "/usr/lib/python3.11/urllib/request.py", line 1348, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/usr/lib/python3.11/http/client.py", line 1294, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.11/http/client.py", line 1340, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.11/http/client.py", line 1289, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.11/http/client.py", line 1048, in _send_output
    self.send(msg)
  File "/usr/lib/python3.11/http/client.py", line 986, in send
    self.connect()
  File "/usr/lib/python3.11/http/client.py", line 1466, in connect
    self.sock = self._context.wrap_socket(self.sock,
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/ssl.py", line 517, in wrap_socket
    return self.sslsocket_class._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/ssl.py", line 1108, in _create
    self.do_handshake()
  File "/usr/lib/python3.11/ssl.py", line 1383, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1006)

Environment

Aria Operations 8.18.x

Cause

Expired Web certificate in Aria Ops prevents the new Data Node from establishing a trusted SSL connection with the Primary node during the initial configuration handshake.

Resolution

Regenerate and install the Aria Ops Web Certificate.

Make sure to take snapshot of the Aria Ops nodes with the process mentioned in below document,

https://knowledge.broadcom.com/external/article/342576/how-to-take-a-snapshot-of-vmware-aria-op.html

 

If you already have Aria Ops cluster configured with multiple nodes, we will need to take cluster offline before changing the certificate using below steps,

• In a Web browser, navigate to the VCF Operations administration interface at https://vcfops-node-FQDN-or-ip-address/admin.
• Log in with the local admin username and password.
• If you are on VCF Operations 8.10 or later, click Take Offline under Cluster Status.

         Note: Wait for Cluster Status to show as Offline.

• If you are on VCF Operations 8.10 or later, once the certificate is installed, click Bring Online under Cluster Status.

         Note: Wait for Cluster Status to show as Online.

Use below KBs depending on whether the certificate is self signed/Custom.

Reload Default certificate

https://knowledge.broadcom.com/external/article/326393/reload-the-default-certificate-in-vreali.html

 

Replace with CA signed certificate,

https://knowledge.broadcom.com/external/article?legacyId=2046591

 

After successfully renewing/installing certificate re-deploy the Data node.