VCF CAP upgrade failure fails to resume

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

CAP upgrade restarts from beginning instead of resuming from previous failure stage. You would see the below errors in third party logs:

2022-10-19 18:50:31,368: INFO:
2022-10-19 18:50:31,369: INFO: RC: 1, OUT:
2022-10-19 18:50:31,369: INFO: ERR: Traceback (most recent call last):
File "/var/log/vmware/vcf/lcm/thirdparty/bundles/b984c5e4-####-####-####-##########a9/thirdparty/cap-platform-setup/bin/cap_platform_setup.py.copy", line 270, in <module>
migration_workflow()
File "/var/log/vmware/vcf/lcm/thirdparty/bundles/b984c5e4-####-####-####-##########a9/thirdparty/cap-platform-setup/bin/cap_platform_setup.py.copy", line 185, in migration_workflow
wrapper.execute_cmd_locally("lsblk -o NAME,TYPE,MOUNTPOINT -n -i -r | grep lvm_snapshot")
File "/var/log/vmware/vcf/lcm/thirdparty/bundles/b984c5e4-####-####-####-##########a9/thirdparty/cap-platform-setup/bin/../../wrapper.py", line 192, in execute_cmd_locally
self.update_status(return_code=rc, status='COMPLETED_WITH_FAILURE', errmsg=err)
File "/var/log/vmware/vcf/lcm/thirdparty/bundles/b984c5e4-####-####-####-##########a9/thirdparty/cap-platform-setup/bin/../../wrapper.py", line 168, in update_status
raise Exception
Exception
2022-10-19 18:50:31,369: ERROR: vcf_platform_upgrade.py trigger script exited with return code 1

 

Environment

Vmware Cloud Foundation 4.5

VMware Cloud Foundation 5.x

Cause

When VCF upgrade fails at CAP Upgrade stage, it fails to resume on retry due to left over CAP upgrade files from previous failure.

Resolution

To resolve the issue, run CAP cleanup workflow prior to resuming upgrade.

Follow the below mentioned steps to run CAP cleanup:

• Get an SSH session to the SDDC Manager and switch user to root.
• Remove vcf-cap-worklfows.json file 
• rm /var/log/vmware/capengine/vcf-cap-workflows.json
• Register a user credential with the following shell commands:
• echo -n <password> | /usr/lib/vmware-capengine/scripts/user_registry.sh add -u apiuser
• authHeader="Basic $(echo -n "apiuser:<password>" | base64)"
• echo $authHeader

This will output the encoded credential.

 

Sample output:

• Start the CAP cleanup workflow:

curl --location --request POST 'http://127.0.0.1:15051/capengine/api/v1/workflow/execute' \

--header 'Content-Type: application/json' \

--header "Authorization: $authHeader" \

--data-raw '{

"workflowName": "cap-update-cleanup",

"operation": "start"

}' | jq

 

Sample Output:

• Re-trigger upgrade from SDDC UI.