The vCenter upgrade encountered a failure with the error: "Error encountered while upgrading SPS data."

Products

• VMware SDDC Manager
• VMware vCenter Server

Issue/Introduction

Symptoms:

• The vCenter upgrade fails during Stage 2 while attempting to import SPS data.

• The logs indicate errors similar to the following:

  • /var/log/vmware/upgrade/import-upgrade-runner.log

yyyy-mm-ddThh:mm:ss.Z INFO upgrade.states.component_states sps:Import: yyyy-mm-ddThh:mm:ss.Z ERROR sps SPS external upgrade failed with a generic error
yyyy-mm-ddThh:mm:ss.Z INFO upgrade.states.component_states sps:Import: yyyy-mm-ddThh:mm:ss.Z ERROR __main__ Upgrade Phase 'sps:Import' failed. Exception: Error encountered while upgrading SPS data
yyyy-mm-ddThh:mm:ss.Z ERROR upgrade.states.component_states sps:Import: Remote script failed with an error Error encountered while upgrading SPS data
yyyy-mm-ddThh:mm:ss.Z ERROR upgrade.states.component_states sps:Import: failed with permanent error. For details take a look at Import_com.vmware.sps_yyyy_mm_dd_hh_4mm.log.
yyyy-mm-ddThh:mm:ss.Z WARNING wf.wf_processor State Import com.vmware.sps has failed.
yyyy-mm-ddThh:mm:ss.Z ERROR wf.wf_processor Workflow execution has failed
yyyy-mm-ddThh:mm:ss.Z ERROR UpgradeRunner Upgrade mode import failed

• • /var/log/vmware/upgrade/Import_com.vmware.sps_yyyy_mm_dd_hh_mm.log

yyyy-mm-ddThh:mm:ss.Z [main] ERROR opId= com.vmware.vim.storage.common.kv.KvManager - Error occured while putting multiple entries in KV store
com.vmware.vim.binding.vmodl.fault.SystemError: InvalidState
..
Caused by: com.vmware.vim.binding.vmodl.fault.SystemError: InvalidState
yyyy-mm-ddThh:mm:ss.Z [main] INFO  opId= com.vmware.sps.UpgradeMain - [main] SPS upgrade completed with exit code : 3
yyyy-mm-ddThh:mm:ss.Z ERROR sps SPS external upgrade failed with a generic error
yyyy-mm-ddThh:mm:ss.Z ERROR __main__ Upgrade Phase 'sps:Import' failed. Exception: Error encountered while upgrading SPS data

• • /var/log/vmware/vpostgres/postgresql.log

yyyy-mm-ddThh:mm:ss UTC 674686db.5121 0 VCDB vc 127.0.0.1(45214) 20769 3 ERROR:  integer out of range
yyyy-mm-ddThh:mm:ss UTC 674686db.5121 0 VCDB vc 127.0.0.1(45214) 20769 5 ERROR:  integer out of range

• Executing the following PostgreSQL query on the VCSA gives a value higher than 2,147,483,647 -

  • psql -d VCDB -U postgres -c "select nextval(pg_get_serial_sequence('VC.CIS_KV_KEYVALUE', 'surr_key'));"


Environment

VMware Cloud foundation
VMware vCenter Server

Cause

• This is a known issue affecting the vCenter Server database where surr_key values become very fragmented to the point that the current value of the sequence reaches the maximum integer value of 2,147,483,64.
• Any subsequent attempt to insert a value into the surr_key column results in an "integer out of range" error.

Resolution

• Currently, there is no resolution to the issue. This will be resolved in future releases.

Workaround:

IMPORTANT: Ensure that fresh snapshots/backups have been taken before any modifications.

• Download the SQL script "defrag_surr_key_cis_kv_keyvalue.sql" attached to this article.
• Use a file-moving utility of your choice (WinSCP for example) to copy the script to the vCenter.
• Note: If you have issues connecting to the vCenter please see - Error when uploading files to vCenter Server Appliance using WinSCP
• Connect to VCSA on SSH while logging in as a root user.
• Change your directory to the location of the file.
• Apply the correct permission to the file:
  • chmod 755 defrag_surr_key_cis_kv_keyvalue.sql
• Execute the script using the following command: 
  • psql -d VCDB -U postgres -f defrag_surr_key_cis_kv_keyvalue.sql
• Restart vCenter services:
  • service-control --stop --all && service-control --start --all
• Reattempt the vCenter server upgrade.