The vCenter Server upgrade from version 7.0 to 8.0 fails at 39% with the error: "Upgrade phase timed out" after getting stuck during the "Exporting the VMware Analytics Service data" step.

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• During Stage 2 of the upgrade process, the task "Exporting VMware Analytics Service data" gets stuck at 39%.
  
  
  • 

 

• Upgrade then fails with the below error:
  
  Upgrade phase timed out. The time planned for the upgrade phase was 60 minutes. The upgrade phase has already been running for 60 minutes. To extend the default timeout, set environment variable UPGRADE_EXPORT_TIMEOUT
  • 

Environment

VMware vCenter Server 7.x
VMware vCenter Server 8.x

Cause

This issue occurs because the push telemetry process sends an excessive number of requests with unique IDs, resulting in the creation of a large number of files in the "/var/log/vmware/analytics/prod" directory.
In some cases, this has led to the accumulation of 185,000 to 375,000 files.

Resolution

1. Connect to the source vCenter server via SSH
2. Run the following command to check the number of files created on the vCenter server due to push telemetry process: ls /var/log/vmware/analytics/prod | wc -l
3. Download the cleaning_vddk_script.sh script from the attachments section of the KB.
4. Upload the script to vCenter and assign "execute" permissions with this command: chmod +x cleaning_vddk_script.sh
5. Run the script: ./cleaning_vddk_script.sh
6. Restart vCenter upgrade.

Note:
By default, the script will apply changes to files older than 180 days. You can however pass a parameter to the script in order to use a different value (Example: 90 days).
./cleaning_vddk_script.sh 90

 

Additional Information

• VMware vCenter Analytics Service is restarting often with out of memory errors