Error: LCMVMSP10002 during upgrade to VCF Automation 9.0.2 due to ebs-app liquibase database migration timeout

Products

• VCF Automation

Issue/Introduction

• Upgrade from VMware Aria Automation 8.18.1 to VCF Automation (VCFA) 9.0.2 fails with error code LCMVMSP10002 : Failed to deploy product on the application platform. Product deployment failed.
• The ebs-app pod enters a continuous CrashLoopBackOff state due to readiness probe failures.
• The ebs-app logs reveals that a Liquibase database migration script (event-broker-service-053.xml) failed while executing an ALTER TABLE ebs_audit_log ADD COLUMN statement designed to add text search capabilities.
  

  Caused by: liquibase.exception.UnexpectedLiquibaseException: liquibase.exception.CommandExecutionException: liquibase.exception.LiquibaseException: liquibase.exception.MigrationFailedException: Migration failed for changeset db/changelog/event-broker-service-053.xml::053-001::######:
  Reason: liquibase.exception.DatabaseException: ERROR: canceling statement due to statement timeout [Failed SQL: (0) ALTER TABLE ebs_audit_log ADD COLUMN ebs_audit_log_search tsvector GENERATED ALWAYS AS (to_tsvector('english', coalesce(message, '') || ' ' || coalesce(entity_name, ''))) STORED]
  at liquibase.integration.spring.SpringLiquibase.afterPropertiesSet(SpringLiquibase.java:259) ~[liquibase-core-4.27.0.jar:na]

Environment

VCF Automation 9.x

Cause

VCF Automation 9.x enforces a strict 30-second database statement timeout. In environments with a large ebs_audit_log table, the ALTER TABLE operation required for adding text search capabilities exceeds this 30-second window, causing the PostgreSQL to explicitly cancel the statement and crash the ebs-app pod.

 

Resolution

To complete the upgrade, the database and connection timeouts must be temporarily extended to allow the large table migration to finish. 

Note: This workaround should be applied after the VCFA upgrade is initiated and the ebs pod fails to start.

SSH using vmware-system-user to one of the VCF Automation node. Switch user to root : sudo su - and follow the below steps  :

Step 1. Increase postgres statement timeout to 5 minutes (300s) :

Connect to the database and extend the statement timeout from 30s to 5 minutes (300 seconds):

base64 -d <<< "a3ViZWN0bCAtbiBwcmVsdWRlIGV4ZWMgJChrdWJlY3RsIGV4ZWMgLW4gcHJlbHVkZSB2Y2ZhcG9zdGdyZXMtMCAtLSBwYXRyb25pY3RsIGxpc3QgMj4vZGV2L251bGwgfCBhd2sgJy9MZWFkZXIvIHtwcmludCAkMn0nKSAtLSBwc3FsIC1VIHBvc3RncmVzIC1jICJTSE9XIHN0YXRlbWVudF90aW1lb3V0OyIgXAogIC1jICJBTFRFUiBTWVNURU0gU0VUIHN0YXRlbWVudF90aW1lb3V0ID0gJzMwMHMnOyIgXAogIC1jICJTRUxFQ1QgcGdfcmVsb2FkX2NvbmYoKTsi" | bash

Step 2. Increase JDBC Socket Timeout :

Edit the deployment configuration of the ebs-app to set the socket timeout to 5 minutes:

# Take backup of ebs-app deployment :
kubectl get deploy ebs-app -n prelude -o yaml > ebs-app-backup.yaml

# Edit EBS deployment :
kubectl edit deploy ebs-app -n prelude

# Add the following property under JAVA_OPTS :
-Dspring.datasource.hikari.data-source-properties.socketTimeout=300 

# Save the change and exit the editor :
:wq! 

Step 3. Restart the EBS service :

Scale the deployment down and up to apply changes and let the Liquibase migration finish:

kubectl scale deployment ebs-app --replicas=0 -n prelude
kubectl scale deployment ebs-app --replicas=$(kubectl get nodes --no-headers | wc -l) -n prelude

(If any dependent pods fail or fail to clear, restart them as well once ebs-app is stable).

Step 4. Wait for all the prelude pods to start :

kubectl get pods -A

Step 5.  Revert to Default Settings:

Once the environment finishes the upgrade configuration and returns to a completely stable state, revert the connection limits to prevent permanent over-extended timeouts:

base64 -d <<< "a3ViZWN0bCAtbiBwcmVsdWRlIGV4ZWMgJChrdWJlY3RsIGV4ZWMgLW4gcHJlbHVkZSB2Y2ZhcG9zdGdyZXMtMCAtLSBwYXRyb25pY3RsIGxpc3QgMj4vZGV2L251bGwgfCBhd2sgJy9MZWFkZXIvIHtwcmludCAkMn0nKSAtLSBwc3FsIC1VIHBvc3RncmVzIC1jICJTSE9XIHN0YXRlbWVudF90aW1lb3V0OyIgXAogIC1jICJBTFRFUiBTWVNURU0gU0VUIHN0YXRlbWVudF90aW1lb3V0ID0gJzMwcyc7IiBcCiAgLWMgIlNFTEVDVCBwZ19yZWxvYWRfY29uZigpOyI=" | bash

Edit the EBS deployment config to remove the socket timeout :

# Edit EBS deployment
kubectl edit deploy ebs-app -n prelude 

#Remove the following property
 -Dspring.datasource.hikari.data-source-properties.socketTimeout=300

# Save the configuration
:wq!

 

Additional Information

The ideal recommended size of audit_log table in Aria Automation 8.18.1 is around 2M. The issue can be prevented from occurring by modifying the database schema change. Contact Broadcom Support and mention KB 445104 on the case description for applying the fix prior to the upgrade attempt.