Upgrading from VCSA 5.x to 6.0 fails with no specific error
search cancel

Upgrading from VCSA 5.x to 6.0 fails with no specific error

book

Article ID: 321335

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms:
  • Upgrading the vCenter Server Appliance 5.x to 6.0 fails without a specific error message
  • In the /var/log/vmware/upgrade/export-upgrade-runner.log file, you see entries similar to:

    <YYYY-MM-DD><time> INFO upgrade.states.component_states is:Export: SEVERE: Unable to login to Inventory Service
    <YYYY-MM-DD><time> INFO upgrade.states.component_states is:Export: Date_Time com.vmware.vim.vmomi.client.http.impl.HttpConfigurationCompilerBase$ConnectionMonitorThreadBase shutdown
    <YYYY-MM-DD><time> INFO upgrade.states.component_states is:Export: WARNING: Shutting down the connection monitor.
    <YYYY-MM-DD><time> INFO upgrade.states.component_states is:Export: </time></time></time></time>
    Date_Time com.vmware.vim.dataservices.BackupProductData backup
    <YYYY-MM-DD><time> INFO upgrade.states.component_states is:Export: SEVERE: Backup PBM failed
    <YYYY-MM-DD><time> INFO upgrade.states.component_states is:Export: java.lang.Exception: Unable to login to Inventory Service
    <YYYY-MM-DD><time> INFO upgrade.states.component_states is:Export: at com.vmware.vim.dataservices.Utils.getTicketAsSuperUser(Utils.java:375)
    <YYYY-MM-DD><time> INFO upgrade.states.component_states is:Export: at com.vmware.vim.dataservices.Utils.loginWithIS(Utils.java:309)
    <YYYY-MM-DD><time> INFO upgrade.states.component_states is:Export: at com.vmware.vim.dataservices.BackupProductData.backup(BackupProductData.java:167)
    <YYYY-MM-DD><time> INFO upgrade.states.component_states is:Export: at com.vmware.vim.dataservices.BackupProductData.main(BackupProductData.java:353)
    <YYYY-MM-DD><time> INFO upgrade.states.component_states is:Export: Caused by: java.util.concurrent.ExecutionException: (vmodl.fault.HostCommunication) {
    <YYYY-MM-DD><time> INFO upgrade.states.component_states is:Export: faultCause = null,
    <YYYY-MM-DD><time> INFO upgrade.states.component_states is:Export: faultMessage = null
    <YYYY-MM-DD><time> INFO upgrade.states.component_states is:Export: }
    <YYYY-MM-DD><time> INFO upgrade.states.component_states is:Export: at com.vmware.vim.vmomi.core.impl.BlockingFuture.get(BlockingFuture.java:70)
    <YYYY-MM-DD><time> INFO upgrade.states.component_states is:Export: at com.vmware.vim.clientutil.InventoryClientUtil.acquireSessionTicket(InventoryClientUtil.java:128)
    <YYYY-MM-DD><time> INFO upgrade.states.component_states is:Export: at com.vmware.vim.clientutil.InventoryClientUtil.getAuthDataForDataService(InventoryClientUtil.java:95)
    <YYYY-MM-DD><time> INFO upgrade.states.component_states is:Export: at com.vmware.vim.dataservices.Utils.getTicketAsSuperUser(Utils.java:370)
    <YYYY-MM-DD><time> INFO upgrade.states.component_states is:Export: ... 3 more
    <YYYY-MM-DD><time> INFO upgrade.states.component_states is:Export: Caused by: (vmodl.fault.HostCommunication) {
    <YYYY-MM-DD><time> INFO upgrade.states.component_states is:Export: faultCause = null,
    <YYYY-MM-DD><time> INFO upgrade.states.component_states is:Export: faultMessage = null
    <YYYY-MM-DD><time> INFO upgrade.states.component_states is:Export: }
    [...]
    <YYYY-MM-DD><time> ERROR upgrade.states.component_states is:Export: Remote script failed with an error [PermanentError()]
    <YYYY-MM-DD><time> ERROR upgrade.states.component_states is:Export: failed with permanent error. For details take a look at Export_com.vmware.is_xxxx_xx_xx_xx_xx.log.</time></time></time></time></time></time></time></time></time></time></time></time></time></time></time></time></time></time></time></time></time>

     
  • In the /var/log/vmware/upgrade/Export_com.vmware.is_xxxx_xx_xx_xx_xx_xx.log file referenced in the previous symptom, you see similar to:

    <DATE><time> com.vmware.vim.dataservices.BackupProductData main SEVERE: Backup failed
    <DATE><time> com.vmware.vim.dataservices.BackupProductData main INFO: Exiting with code 1 after Backup Product.
    <YYYY-MM-DD><time> ERROR __main__ Upgrade Phase 'is:Export' failed. Exception: Error attempting Backup PBM Traceback (most recent call last):
    File "/tmp/vmware-root-4013919865/tmpvmware60/payload/componentPhaseLauncher.py", line 379, in main executionResult = systemExtension(exeContext)
    File "/tmp/vmware-root-4013919865/tmpvmware60/libs/sdk/extensions.py", line 94, in __call__result = self.extension(*args)
    File "/tmp/vmware-root-4013919865/tmpvmware60/libs/sdk/extensions.py", line 110, in _func return func(*args)
    File "/tmp/vmware-root-4013919865/tmpvmware60/payload/component-scripts/is/__init__.py", line 564, in exportData execIsCmd(reporter, backUpPBM, "Backup PBM", 20, sourceComponentAssistant)
    File "/tmp/vmware-root-4013919865/tmpvmware60/payload/component-scripts/is/__init__.py", line 240, in execIsCmd raiseUpgradeFailed(opStr)
    File "/tmp/vmware-root-4013919865/tmpvmware60/payload/component-scripts/is/__init__.py", line 222, in raiseUpgradeFailed raise PermanentError(cause=errStr, resolution=resStr)
    PermanentError: Error attempting Backup PBM
    </time></time></time>
​​Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.


Environment

VMware vCenter Server Appliance 6.0.x
VMware vCenter Server Appliance 5.5.x

Cause

This issue occurs when:
  • There is an incorrect certificate replacement through the Management Interface on the vCenter Server Appliance.
  • If you used the option Allow Certificate Generation on the Port 5480 Admin Interface on the vCenter Server Appliance 5.x, the connection between vCenter Server and Inventory Service gets lost.

Resolution

To resolve this issue, remove the association of the vCenter Server service and Inventory Service.
  1. Connect to the vCenter Server Appliance with an SSH or Console session using root credentials.
  2. Move the mv /var/vmware/vpxd/inventoryservice_registered /tmp/ file to a different location to clear the association of vCenter Server service and the Inventory Service.
  3. Restart the vCenter Server Appliance Service:

    service vmware-vpxd restart


Additional Information

Re-pointing and re-registering VMware vCenter Server Appliance 5.5.x and component