Aria Automation UI Inaccessible Due to vCO Container Startup Failure
search cancel

Aria Automation UI Inaccessible Due to vCO Container Startup Failure

book

Article ID: 390627

calendar_today

Updated On:

Products

VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

  • Aria Automation (VRA) fails to start after a reboot or service restart using /opt/scripts/deploy.sh.
  • The deploy.sh script fails due to the vCO pod taking more than 30 minutes to start.
  • The vCO pod shows as "Running" but with a status of "2/3 Ready".

NAMESPACE     NAME                                              READY   STATUS      RESTARTS      AGE

prelude       vco-app-d68b78948-9fgd9                           2/3     Running     0             37d

  • The /var/log/vmware/prelude/helm/prelude_vco.log contains messages indicating the vCO startup exceeded the Kubernetes health probe timeout.

[2025-02-28 12:02:04] DEBUG Running command `kubectl get deployment vco-app --namespace prelude -o json`
[2025-02-28 12:02:04] DEBUG Retrying helm_wait_check after 32.22 seconds; 85 failed attempts
[2025-02-28 12:02:37] DEBUG Running command `helm status vco --namespace prelude --output json`
[2025-02-28 12:02:37] INFO Command `helm status vco --namespace prelude --output json` stderr: WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/admin.conf
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /etc/kubernetes/admin.conf

[2025-02-28 12:02:37] DEBUG Running command `kubectl get deployment vco-app --namespace prelude -o json`
[2025-02-28 12:02:37] ERROR Release 'vco' in namespace 'prelude' failed to come up
Traceback (most recent call last):
  File "/opt/scripts/helm-upstall", line 286, in main
    helm_wait(namespace, release_name, timeout=timeout)
  File "/opt/scripts/helm-upstall", line 127, in helm_wait
    helm_wait_check()
  File "/opt/python-modules/vracli/decorators.py", line 161, in wrapper
    raise RuntimeError(f'retry timeout exceeded for {f_name}') from err
RuntimeError: retry timeout exceeded for helm_wait_check

    • The vCO server log /services-logs/prelude/vco-app/file-logs/vco-server-app.log shows errors related to loading WSDL and SOAP service availability.

2025-02-28T11:07:00.763Z ERROR vco [host='vco-app-5bb4bdbb55-vkgll' thread='ApplicaitonEventHandler-1' user='' org='' trace=''] {} com.vmware.o11n.plugin.soap.axis2.Axis2SOAPAdaptor - Unable to load content of the WSDL and its schemas for '258dc25a-2991-4c96-b88b-a5f132c85301', WSDL content:
2025-02-28T11:07:00.771Z ERROR vco [host='vco-app-5bb4bdbb55-vkgll' thread='ApplicaitonEventHandler-1' user='' org='' trace=''] {} com.vmware.o11n.plugin.soap.SOAPHost - Axis service is NOT available!

2025-02-28T11:00:27.545Z DEBUG vco [host='vco-app-5bb4bdbb55-vkgll' thread='ApplicaitonEventHandler-1' user='' org='' trace=''] {} org.apache.commons.httpclient.HttpMethodDirector - Closing the connection.
2025-02-28T11:00:27.545Z INFO vco [host='vco-app-5bb4bdbb55-vkgll' thread='ApplicaitonEventHandler-1' user='' org='' trace=''] {} org.apache.commons.httpclient.HttpMethodDirector - I/O exception (java.net.ConnectException) caught when processing request: Connection timed out
2025-02-28T11:00:27.545Z DEBUG vco [host='vco-app-5bb4bdbb55-vkgll' thread='ApplicaitonEventHandler-1' user='' org='' trace=''] {} org.apache.commons.httpclient.HttpMethodDirector - Connection timed out
java.net.ConnectException: Connection timed out
        at sun.nio.ch.Net.connect0(Native Method) ~[?:?]


2025-02-28T11:00:27.547Z INFO vco [host='vco-app-5bb4bdbb55-vkgll' thread='ApplicaitonEventHandler-1' user='' org='' trace=''] {} org.apache.commons.httpclient.HttpMethodDirector - Retrying request
2025-02-28T11:00:27.547Z DEBUG vco [host='vco-app-5bb4bdbb55-vkgll' thread='ApplicaitonEventHandler-1' user='' org='' trace=''] {} org.apache.commons.httpclient.HttpConnection - Open connection to FQDN-OF-SOAP-HOST:7003

Environment

Aria Automation 8.X

Aria Orchestrator 8.x

Cause

Setting an excessively long connection timeout (e.g., 30 minutes) for the SOAP plugin host can significantly delay VRO startup and cause the entire deployment to fail.

 

Resolution

The error messages indicate that the vCO is unable to reach the SOAP endpoint. This is likely due to a misconfigured or unavailable SOAP host. To resolve this:

  1. Remove Unreachable SOAP Host from vCO Database:

    • Login to the vCO database using vracli dev psql vco-db.
    • Delete the configuration for the unreachable SOAP host using the following SQL command:
      delete from vmo_resourceelement where name = '258dc25a-2991-4c96-b88b-a5f132c85301';
    • Exit the database using \q.

  2. Restart VRA Services:

    • Run the /opt/scripts/deploy.sh script to restart VRA services.

  3. Investigate and Re-add SOAP Host:

    • Once VRA and VRO are up and running, investigate why the SOAP host was inaccessible. This could involve verifying network connectivity, firewall rules, or the status of the SOAP service itself.
    • Once the issue is resolved, re-add the SOAP host to the vCO configuration. Refer to the vCO documentation for specific instructions on adding SOAP hosts.

Note: The specific identifier 258dc25a-2991-4c96-b88b-a5f132c85301 represents the vCO configuration element for the unreachable SOAP host. Replace this with the actual identifier found in your vCO logs.

Powered by Wolken Software