Upgrade to VCF Operations 9.1 fails at step 9 of 14 - run master postgres db upgrade

Products

• VCF Operations

Issue/Introduction

To determine if this KB applies for the VCF Operations 9.1 upgrade failure:

1. Log in to the VCF Operations Admin UI and navigate to the Software Update tab

   1. Validate the Steps Completed column for the primary node displays "9 of 14 - Applied Operating System Update"

   2. Validate that the Status column for the primary node displays "Failed resource key=pak_manager.action_failed, resource args=[run master postgres db upgrade]"

      
      
      

2. Log in to the primary node as root via SSH or vSphere Console

   1. Validate the database version

      su - postgres -c "/opt/vmware/vpostgres/current/bin/psql -d vcopsdb -p 5433 -c 'select * from version;'"

      The version should match exactly

       major | minor | build | version_id
      -------+-------+-------+------------
           1 |     0 |   976 |          1

   2. Check for entries that will cause foreign key errors with the upgrade script

      su - postgres -c "/opt/vmware/vpostgres/current/bin/psql -d vcopsdb -p 5433 -c 'SELECT * FROM vcf_entitlement_policy_warning AS vepw WHERE NOT EXISTS (SELECT 1 FROM vcf_entitlement AS ve WHERE ve.allocation_id = vepw.allocation_id);'"

      Validate at least 1 entry is returned.

If all steps above match, proceed to the resolution section.

Environment

• VCF Operations 9.0.x
• VCF Operations 9.1

Cause

The scripts that perform the database upgrade exit due a foreign key error in specific cases.

Resolution

Contact Broadcom VCF Support and reference this KB number.

Additional Information

When this issue occurs, the following error will appear in /storage/vcops/log/pakManager/vRealizeOperationsManagerEnterprise-910025346023/apply_system_update_stderr.log

COMMIT;]; ERROR: insert or update on table "vcf_entitlement_policy_warning" violates foreign key constraint "vcf_entitlement_policy_warning_allocation_id_fkey"
Detail: Key (allocation_id, license_manager_asset_id)=(xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, ########-####-####-####-############) is not present in table "vcf_entitlement".
Where: SQL statement "ALTER TABLE vcf_entitlement_policy_warning