SDDC Manager upgrade fails at the stage "Update VCF Service and Platform rpms"

Products

• VMware SDDC Manager
• VCF - MCOE - SDDC

Issue/Introduction

In the log file /var/log/vmware/vcf/lcm/thirdparty/upgrades/<upgrade-id>/vcf-platform/upgrade/vcf_platform_upgrade.log on the SDDC Manager, we see error entries:

YYYY-MM-DDTHH:MM:SS,227: INFO: URL: http://###.##.##15051/capengine/api/v1/workflow/instance/######-####-####-####-#########
YYYY-MM-DDTHH:MM:SS,233: ERROR: RC: 500, OUT: {'errorMessage': 'Workflow instance######-####-####-####-######### not found'} ERR:
NoneType: None
YYYY-MM-DDTHH:MM:SS,233: ERROR: 'status'
Traceback (most recent call last):
File "/var/log/vmware/vcf/lcm/thirdparty/bundles/######-####-####-####-#########/thirdparty/vcf-platform-upgrade/bin/vcf_platform_upgrade.py.copy", line 472, in <module>
upgrade_workflow()
File "/var/log/vmware/vcf/lcm/thirdparty/bundles/######-####-####-####-#########/thirdparty/vcf-platform-upgrade/bin/vcf_platform_upgrade.py.copy", line 116, in upgrade_workflow
if cap_update_status["status"] == 'Failed':
KeyError: 'status'

Environment

VMware Cloud Foundation 4.5.x

VMware Cloud Foundation 5.0.x

VMware Cloud Foundation 5.1.x

VMware Cloud Foundation 5.2.0

VMware Cloud Foundation 5.2.1

Cause

The issue is caused by a bug in the sqllite3 Go library, triggered when the select query returns no rows for a particular ID during concurrent operations in the database.

Resolution

The sqllite3 library was reconfigured and the issue is resolved in VCF 5.2.1

Workaround:

1. SSH into the SDDC Manager and monitor the log file /var/log/vmware/capengine/cap-update/workflow.log until the workflow finishes:

   YYYY-MM-DDTHH:MM:SS.426528 workflow_manager.go:221: Task stage-cleanup completed
YYYY-MM-DDTHH:MM:SS.426646 workflow_manager.go:183: All tasks finished for workflow
YYYY-MM-DDTHH:MM:SS.426699 workflow_manager.go:354: Updating instance status to Completed

2. Reboot the SDDC Manager VM after the upgrade failure:
   
   SSH into SDDC Manager VM as VCF user
   and run the command "reboot"

3. Configure FIPS for Java:

   mkdir -p /usr/lib/jvm/openjdk-java17-headless.x86_64/conf/security;

   cp -rf /var/log/vmware/vcf/lcm/thirdparty/bundles/<sddc-manager-bundle-id>/thirdparty/vcf-platform-upgrade/lib/java.fips.security /usr/lib/jvm/openjdk-java17-headless.x86_64/conf/security/java.security;

   chmod 644 /usr/lib/jvm/openjdk-java17-headless.x86_64/conf/security/java.security

4. Copy Java configuration:

   mkdir -p /usr/lib/systemd/system.conf.d/;

   cp -rf /var/log/vmware/vcf/lcm/thirdparty/bundles/<sddc-manager-bundle-id>/thirdparty/vcf-platform-upgrade/lib/java-01-javabcapproved.conf /usr/lib/systemd/system.conf.d/01-javabcapproved.conf;

   chmod 644 /usr/lib/systemd/system.conf.d/01-javabcapproved.conf

5. Copy BouncyCastle FIPS libs to the classpath:

   mkdir -p /opt/vmware/vcf/fips; chmod 755 /opt/vmware/vcf/fips;

   cp -rf /var/log/vmware/vcf/lcm/thirdparty/bundles/<sddc-manager-bundle-id>/thirdparty/vcf-platform-upgrade/lib/bc-fips-1.0.2.4.jar /opt/vmware/vcf/fips/;

   chmod 644 /opt/vmware/vcf/fips/bctls-fips-1.0.16.jar;

   chmod 644 /opt/vmware/vcf/fips/bc-fips-1.0.2.4.jar

6. Reboot the VM:

   sh -x /var/log/vmware/vcf/lcm/thirdparty/bundles/<sddc-manager-bundle-id>/thirdparty/reboot_script.sh

7. Continue the SDDC Manager upgrade workflow.