VMware Aria Automation Orchestrator 8.x services to fail to start or the Monitoring client logs are not visible in the GUI

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

• VMware Aria Automation Orchestrator 8.x (formerly vRealize Orchestrator) and Monitoring client logs are not visible in the GUI.
  • This is likely to occur on upgrades.
• For 7.x versions, the /var/log/vmware/vco/app-server/server.log,you see entries similar to:

  [ForkJoinPool-47-worker-1] WARN  {} [StubbornRetrier] Retry 6/6 failed because of: Could not load codec 'Lucene60'.  Did you forget to add lucene-backward-codecs.jar?
  [http-nio-127.0.0.1-8280-exec-2] WARN  {} [FileLoggingServiceImpl] Unable to retrieve previous logs:
  java.lang.RuntimeException: Cannot succeed after 6 retries.

• In some scenarios, VMware Aria Automation Orchestrator 8.x services fail to start.
  • /opt/scripts/deploy.sh times out while waiting on services to start.
• When inspecting vco-app-id pod status with command kubectl -n prelude get pods Automation Orchestrator services are reported unhealthy with 2 out of 3 containers deployed; running tail against the /data/vco/usr/lib/vco/app-server/logs/catalina.log file shows that the server hangs at the step below:

  xx-xxx-20xx 16:03:24.980 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/usr/lib/vco/app-server/deploy/vco.war]
  xx-xxx-20xx 16:03:28.497 INFO [localhost-startStop-1] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.

Environment

VMware vRealize Orchestrator 8.x
VMware Aria Automation Orchestrator 8.x

Cause

This issue occurs when Log files are indexed with the previous version's Lucene codec. When upgraded, the index is not automatically re-coded with the new Lucene codec.

Resolution

Currently, there is no resolution for this issue. See the Workaround below for further details.

Workaround:

VMware Aria Automation Orchestrator 8.x (formerly vRealize Orchestrator)

To temporarily workaround the issue, delete the log indices and restart the Automation Orchestrator service by deleting the pod by its ID.

Prerequisites

• You have enough available diskspace to perform file deletion.

Procedure

1. On each virtual appliance(s) within the cluster run

   rm -rf /data/vco/usr/lib/vco/app-server/logs/scripting.log*_lucene*

Note: The file can be in format of scripting.log.Number_lucene. A wildcard asterisk ( * ) may be needed if there are multiple lucene files.

2. Isolate the pod ID(s) for the Automation Orchestrator service(s)

   kubectl get pods -n prelude

Note: The service is in the format of: vco-app-randomID-randomID
Example:

3. Delete the pod. This can be done from one appliance, but must be performed for each pod if in a 3-node cluster:

   kubectl delete pods -n prelude pod_id1
   kubectl delete pods -n prelude pod_id2
   kubectl delete pods -n prelude pod_id3

VMware vRealize Orchestrator 7.x

To delete the old Lucene index, run the following commands in a SSH session terminal.

1. Stop vRO 7.x server service:

   service vco-server stop
   

2. Use the find command to isolate the affected logs:

   find /var/log/vco/app-server -iname "*lucene*"
   

3. Delete the old indices:

   rm -rf /var/log/vco/app-server/scripting.log*_lucene*
   

Note:  The file can be in format of scripting.log.Number_lucene. A wildcard asterisk ( * ) may be needed if there are multiple lucene files.

4. Start the vRO 7.x server service:

   service vco-server start