Upgrading VMware Cloud Foundation from 3.0.1 to 3.5.0 fails at sddc-manager-ui-app stage.

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

• You see messages similar to the following in the /var/log/vmware/vcf/lcm/lcm.log file on the SDDC Manager Controller VM

2019-10-29 12:14:56.096 [pool-3-thread-10] INFO [com.vmware.evo.sddc.lcm.services.impl.UpgradeAvailabilityServiceImpl] mainstream bundle 3.1.1-106585 will be available. Rest will be pending
2019-10-29 12:15:00.316 [pool-3-thread-7] INFO [com.vmware.evo.sddc.lcm.orch.Orchestrator] Started orchestrating upgrade 5fb83a7d-2e4e-48fb-81b1-e3d7c760bf83
2019-10-29 12:15:00.591 [ThreadPoolTaskExecutor-5] INFO [com.vmware.evo.sddc.lcm.primitive.evo.util.spi.EvoPrimitiveRestUtil] upgradeId=5fb83a7d-2e4e-48fb-81b1-e3d7c760bf83,resourceType=SDDC_MANAGER_UI,resourceId=51b7284f-3a1c-4d91-828a-167190623d39,bundleElementId=1b6f2bc4-98cb-4de5-9228-cbd7328923ed In EvoPrimitiveRestUtil, getting the upgrade status from http://127.0.0.1/sddc-manager-ui-app/internal/upgrades/5fb83a7d-2e4e-48fb-81b1-e3d7c760bf83
2019-10-29 12:15:00.597 [ThreadPoolTaskExecutor-5] INFO [com.vmware.evo.sddc.lcm.primitive.evo.util.spi.EvoPrimitiveRestUtil] upgradeId=5fb83a7d-2e4e-48fb-81b1-e3d7c760bf83,resourceType=SDDC_MANAGER_UI,resourceId=51b7284f-3a1c-4d91-828a-167190623d39,bundleElementId=1b6f2bc4-98cb-4de5-9228-cbd7328923ed In EvoPrimitiveRestUtil, got upgrade status from http://127.0.0.1/sddc-manager-ui-app/internal/upgrades/5fb83a7d-2e4e-48fb-81b1-e3d7c760bf83 : {"upgradeId":"5fb83a7d-2e4e-48fb-81b1-e3d7c760bf83","resourceId":"51b7284f-3a1c-4d91-828a-167190623d39","startTime":1572351060424,"lastUpdatedTime":1572351060516,"upgradeStatusCode":"INPROGRESS","progress":20}
2019-10-29 12:15:00.597 [ThreadPoolTaskExecutor-5] INFO [com.vmware.evo.sddc.lcm.primitive.evo.impl.VcfServicePrimitiveImpl] upgradeId=5fb83a7d-2e4e-48fb-81b1-e3d7c760bf83,resourceType=SDDC_MANAGER_UI,resourceId=51b7284f-3a1c-4d91-828a-167190623d39,bundleElementId=1b6f2bc4-98cb-4de5-9228-cbd7328923ed In VcfServicePrimitiveImpl, pollUpgradeStatus, received INPROGRESS status while polling SDDC_MANAGER_UI upgrade Status
 

• Timeout received after 30 minutes in /var/log/vmware/vcf/lcm/lcm.log

2019-10-29 12:41:00.807 [ThreadPoolTaskExecutor-5] ERROR [com.vmware.evo.sddc.lcm.primitive.evo.impl.VcfServicePrimitiveImpl] upgradeId=5fb83a7d-2e4e-48fb-81b1-e3d7c760bf83,resourceType=SDDC_MANAGER_UI,resourceId=51b7284f-3a1c-4d91-828a-167190623d39,bundleElementId=1b6f2bc4-98cb-4de5-9228-cbd7328923ed In VcfServicePrimitiveImpl, pollUpgradeStatus, SDDC_MANAGER_UI upgrade timedout
2019-10-29 12:41:00.809 [ThreadPoolTaskExecutor-5] INFO [com.vmware.evo.sddc.lcm.orch.PrimitiveService] upgradeId=5fb83a7d-2e4e-48fb-81b1-e3d7c760bf83,resourceType=SDDC_MANAGER_UI,resourceId=51b7284f-3a1c-4d91-828a-167190623d39,bundleElementId=1b6f2bc4-98cb-4de5-9228-cbd7328923ed primitive upgrade returned status TIMEDOUT
2019-10-29 12:41:01.088 [pool-3-thread-2] INFO [com.vmware.evo.sddc.lcm.orch.Orchestrator] Started orchestrating upgrade 5fb83a7d-2e4e-48fb-81b1-e3d7c760bf83
2019-10-29 12:41:01.090 [pool-3-thread-2] INFO [com.vmware.evo.sddc.lcm.orch.Orchestrator] Upgrade 5fb83a7d-2e4e-48fb-81b1-e3d7c760bf83 reached final state TIMEDOUT
2019-10-29 12:41:01.101 [pool-3-thread-2] INFO [com.vmware.evo.sddc.lcm.events.spi.impl.LcmEventServiceImpl] Creating LCM audit event for notification for eventName: UPGRADE_TIMEDOUT
2019-10-29 12:41:01.102 [pool-3-thread-2] DEBUG [com.vmware.evo.sddc.lcm.events.spi.impl.LcmEventServiceImpl] Before publishing the event: UPGRADE_TIMEDOUT for notification
2019-10-29 12:41:01.102 [pool-3-thread-2] DEBUG [com.vmware.evo.sddc.lcm.events.spi.impl.LcmEventServiceImpl] Successfully published the event: UPGRADE_TIMEDOUT for notification

 

• There are references to the upgrade not starting in /var/log/vmware/vcf/sddc-manager-ui-app/sddc-manager-ui-app.log

{"name":"UI_SERVICE_UPGRADE_PRE_VALIDATION","description":"Upgrade precheck operations","status":"COMPLETED_WITH_SUCCESS","startTime":1572351060424,"endTime":1572351060516,"progress":"100"}
{"name":"UI_SERVICE_UPGRADE","description":"Upgrade the UI server from RPM","status":"NOT_STARTED"}
{"name":"UI_SERVICE_UPGRADE_POST_VALIDATION","description":"Verify install successful","status":"NOT_STARTED"}
 

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Resolution

This is a known issue affecting VMware Cloud Foundation. This issue is resolved in VMware Cloud Foundation 3.8.0.

Workaround:
Use the following steps to workaround the issue, by removing the incomplete upgrade entry from the database 'sddc_manager_ui'.

Login into the SDDC Manager appliance

1. Connect to the SDDC Manager appliance with SSH.
2. Login with the vcf user
3. Change owner to root.

su root

4. Access the Postgres Database with the below command:

psql --host=localhost -U postgres -d sddc_manager_ui

5. Delete the contents of the persistent_state table:

delete from persistent_state;

6. Exit out of the psql database:

\q

7. Manually install sddc-manager-ui-app rpm (the bundle is downloaded in the NFS path)

rpm -Uvh <rpm filepath> /nfs/vmware/vcf/nfs-mount/bundle/<bundleid>/<bundle>/sddc-manager-ui-app-<buildversion >.noarch.rpm

Example:

rpm -Uvh /nfs/vmware/vcf/nfs-mount/bundle/28124dd4-5908-401c-b695-d8681cb98b44/bundle-8203/sddc-manager-ui-app-3.5.1-12050455.noarch.rpm

8. Restart the sddc-manager-ui-app service by running the following command:

systemctl restart sddc-manager-ui-app

9. Resume the Update from the SDDC Manager UI.
10. Login into SDDC Manager UI and continue the upgrade. LCM will check the sddc-manager-ui-app version and mark the sddc-manager-ui-app as updated.