Deploy script fails waiting on VMware Aria Automation Orchestrator 8.x services to start

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

• /opt/scripts/deploy.sh times out while waiting on VMware Aria Automation Orchestrator (formerly VMware vRealize Orchestrator) 8.x services to start.
• When inspecting vco-app-id pod status with command kubectl -n prelude get pods Automation Orchestrator services are reported unhealthy with 2 out of 3 containers deployed; running tail against the /data/vco/usr/lib/vco/app-server/logs/catalina.log file shows that the server hangs at the step below:

  xx-xxx-20xx 16:03:24.980 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/usr/lib/vco/app-server/deploy/vco.war]
  xx-xxx-20xx 16:03:28.497 INFO [localhost-startStop-1] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.

Environment

VMware Aria Automation 8.x
VMware vRealize Automation 8.x
VMware Aria Automation Orchestrator 8.x
VMware vRealize Orchestrator 8.x

Cause

The server will not start until all scripting logs in data/vco/usr/lib/vco/app-server/logs/ are not indexed.
If there are too many logs or large scripting logs, the indexation can take too long (more than an hour) before the server is up and running.

Resolution

VMware Aria Automation Orchestrator 8.x services to fail to start or the Monitoring client logs are not visible in the GUI

Workaround:

Procedure

1. SSH into one VM in the cluster.
2. Run the following command to stop all services:

   /opt/scripts/deploy.sh --shutdown

3. Run the following command to delete all scripting logs from the /data/vco/usr/lib/vco/app-server/logs/ directory:

Note: For clustered environments, this step needs to be executed on each node where these services fail to start.

rm -rf scripting.lo*

4. From the root directory, run the following command to start the deploy script:

   /opt/scripts/deploy.sh