Troubleshooting requests not executing, High CPU usage, and the "Failed to fetch system information message" in VMware Aria Suite Lifecycle

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

• Attempts to access System Details under Home > Settings > System Details fails with message:

  "Failed to fetch system information"

• Other tasks in UI such as log bundle creation and basic day 2 operations do not proceed.
• The CPU usage is high with top command identifying the postgres service as a high consumer.

Environment

VMware vRealize Suite Lifecycle Manager 8.x

Cause

• This issue can occur when there is insufficient storage available for the postgres database or a large number of requests become queued and overload the requests engine.

Resolution

• Prerequisites

• You have valid backups or recent snapshots of the Aria Suite Lifecycle appliance.
• You have access to ssh and the root username and password.

• Procedure

1. Validate that the Aria Suite Lifecycle postgres DB has sufficient space under the db partition /storage. If the disk space is low increase the disk size in vCenter and reboot the Aria Lifecycle appliance. To validate the space login to Aria Lifecycle appliance via ssh and run:

   df -h

2. Stop the engine service
   

   systemctl stop vrlcm-server

3. After validating storage check the database for a large number of stuck requests. To connect to Aria Lifecycle appliance database:

   /opt/vmware/vpostgres/11/bin/psql -U postgres -d vrlcm

4. To query for in-progress requests:

   select count(*) from vm_rs_request where requestname='lcmgenricsetting';
   select count(*) from vm_engine_execution_request where enginestatus='INITIATED';
   select count(*) from vm_engine_statemachine_instance where status='CREATED';
   select count(*) from vm_engine_event where status='IN_PROGRESS';

5. If any of the above commands returns a count equal to or higher than 50 remove the stuck requests:

   delete from vm_rs_request where requestname='lcmgenricsetting';
   delete from vm_engine_execution_request where enginestatus='INITIATED';
   delete from vm_engine_statemachine_instance where status='CREATED';
   delete from vm_engine_event where status='IN_PROGRESS';

6. If a significantly large number of requests have been deleted, vacuum the relevant table:

VACUUM FULL verbose analyze;

7. ​​​​Exit the DB.

\q

9. Start the engine service.

systemctl start vrlcm-server