vCenter upgrade fails with Error: Internal error occurs during Export of VMware vSphere Auto Deploy
search cancel

vCenter upgrade fails with Error: Internal error occurs during Export of VMware vSphere Auto Deploy

book

Article ID: 406654

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • vCenter upgrade fails with below error message in Stage 2.

 

  • In the source vCenter, below log snippets are available:

    • From /var/log/vmware/upgrade/export-upgrade-runner.log:

YYYY-MM-DDTHH:MM:SS INFO upgrade.states.component_states rbd:Export: struct.error: ushort format requires 0 <= number <= (0x7fff * 2 + 1)
YYYY-MM-DDTHH:MM:SS INFO upgrade.states.component_states rbd:Export: STDERR: /tmp/vmware-upgrade-temp-dirHK8n48CJnK/tmpzDlyx41dkd/payload/componentPhaseLauncher.py:86: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
YYYY-MM-DDTHH:MM:SS INFO upgrade.states.component_states rbd:Export: STDERR:   import imp
YYYY-MM-DDTHH:MM:SS INFO upgrade.states.component_states rbd:Export: Script completed for 3.0248632431030273 secs with return-code='1', and executionId=xxxxx-xxxx-xxxx-xxxx-xxxxxxxx
YYYY-MM-DDTHH:MM:SS ERROR upgrade.states.component_states rbd:Export: Remote script failed with an error [InternalError()]
YYYY-MM-DDTHH:MM:SS ERROR upgrade.states.component_states rbd:Export: failed with internal error. For details take a look at Export_com.vmware.rbd_YYYY_MM_DD_HH_MM.log.
YYYY-MM-DDTHH:MM:SS WARNING wf.wf_processor State Export com.vmware.rbd has failed.
YYYY-MM-DDTHH:MM:SS ERROR wf.wf_processor Workflow execution has failed
YYYY-MM-DDTHH:MM:SS WARNING root stopping status aggregation...
YYYY-MM-DDTHH:MM:SS INFO UpgradeRunner Persisting upgrade workflow context..
YYYY-MM-DDTHH:MM:SS INFO deployer.migration_env_deployer Cleaning-up migration environment...
YYYY-MM-DDTHH:MM:SS INFO utils.loginspec_utils User credentials based login specification.
YYYY-MM-DDTHH:MM:SS INFO utils.loginspec_utils The vCenter Server username and password are specified.
YYYY-MM-DDTHH:MM:SS INFO bindings.vc_binding User credentials based authentication.
YYYY-MM-DDTHH:MM:SS INFO deployer.migration_env_deployer The extension 'com.vmware.migrate-connector.127.0.0.1' has been uninstalled
YYYY-MM-DDTHH:MM:SS INFO deployer.migration_env_deployer The components located on 127.0.0.1 has been successfully unregistered as extension.
YYYY-MM-DDTHH:MM:SS INFO deployer.migration_env_deployer Cleaning up target: None target with address=127.0.0.1
YYYY-MM-DDTHH:MM:SS ERROR UpgradeRunner Upgrade mode export failed
YYYY-MM-DDTHH:MM:SS INFO root Exiting with exit-code 1

    • From /var/log/vmware/upgrade/Export_com.vmware.rbd_MMMM_MM_DD_HH_MM.log:

      YYYY-MM-DDTHH:MM:SS INFO service_manager The service vmware-rbd-watchdog has been already stopped
      YYYY-MM-DDTHH:MM:SS INFO service_manager Retrieving status from service 'vmware-rbd-watchdog' ...
      YYYY-MM-DDTHH:MM:SS INFO service_manager Executing command '[['/bin/service-control', '--status', 'vmware-rbd-watchdog']]'
      YYYY-MM-DDTHH:MM:SS INFO service_manager Command '[['/bin/service-control', '--status', 'vmware-rbd-watchdog']]' has exit-code='0' and stdout: Stopped: vmware-rbd-watchdog

      stderr:

      YYYY-MM-DDTHH:MM:SS INFO service_manager Service vmware-rbd-watchdog: STOPPED
      YYYY-MM-DDTHH:MM:SS INFO autodeploy.upgrade_util Successfully wrote the XML changes
      YYYY-MM-DDTHH:MM:SS INFO autodeploy.upgrade_util The config data is {'serviceport': '6501', 'managementport': '6502', 'cacheSize': '2'}
      YYYY-MM-DDTHH:MM:SS ERROR autodeploy.upgrade_util Unable to fetch the logs
      Traceback (most recent call last):
        File "/tmp/vmware-upgrade-temp-dirHK8n48CJnK/tmpzDlyx41dkd/payload/component-scripts/autodeploy/upgrade_util.py", line 530, in fetchLogs
          logZip.write(os.path.join(root_dir, name))
        File "/usr/lib/python3.7/zipfile.py", line 1746, in write
          with open(filename, "rb") as src, self.open(zinfo, 'w') as dest:
        File "/usr/lib/python3.7/zipfile.py", line 1473, in open
          return self._open_to_write(zinfo, force_zip64=force_zip64)
        File "/usr/lib/python3.7/zipfile.py", line 1586, in _open_to_write
          self.fp.write(zinfo.FileHeader(zip64))
        File "/usr/lib/python3.7/zipfile.py", line 447, in FileHeader
          len(filename), len(extra))
      struct.error: ushort format requires 0 <= number <= (0x7fff * 2 + 1)
      YYYY-MM-DDTHH:MM:SS ERROR autodeploy.autodeploy_upgrader Unable to perform a successful export
      Traceback (most recent call last):
        File "/tmp/vmware-upgrade-temp-dirHK8n48CJnK/tmpzDlyx41dkd/payload/component-scripts/autodeploy/autodeploy_upgrader.py", line 210, in exportData
          fetchLogs(metadata, si, sourceVcVersion, exportDir)
        File "/tmp/vmware-upgrade-temp-dirHK8n48CJnK/tmpzDlyx41dkd/payload/component-scripts/autodeploy/upgrade_util.py", line 530, in fetchLogs
          logZip.write(os.path.join(root_dir, name))

    • From /var/log/vmware/rbd:

      -rw-r -- r --    1 root root 30M Mar 13         preUpgradeLogs.zip
      -rw-r -- r --    1 deploy deploy 977K MMM DD    2140 rbd-watchdog-linux. log.5
      -rw-r -- r --    1 deploy deploy 977K MMM DD    2140 rbd-server.log. 9
      -rw-r -- r --    1 deploy deploy 977K MMM DD    2140 rbd-server.log.8
      -rw-r -- r --    1 deploy deploy 977K MMM DD    2140 rbd-watchdog-linux. log.4
      -rw-r -- r --    1 deploy deploy 977K MMM DD    2140 rbd-server.log.7
      -rw-r -- r --    1 deploy deploy 10M  MMM DD     2140 rbd-vmca-certificate.log.2
      -rw-r -- r --    1 deploy deploy 977K MMM DD    2140 rbd-server.log. 6
      -rw-r -- r-      1 deploy deploy 977K MMM DD    2140 rbd-watchdog-linux.log.3
      -rw-r -- r-      1 deploy deploy 977K MMM DD    2140 rbd-server.log.5
      -rw-r -- r --    1 deploy deploy 977K MMM DD    2140 rbd-server.log.4
      -rw-r -- r --    1 deploy deploy 977K MMM DD    2140 rbd-watchdog-linux. log.2
      -rw-r -- r --    1 deploy deploy 977K MMM DD    2140 rbd-server.log.3
      -rw-r -- r --    1 deploy deploy 977K MMM DD    2140 rbd-server.log.2
      -rw-r -- r --    10M                  MMM DD    2140 rbd-vmca-certificate.log.1
      -rw-r -- r --    1 deploy deploy 976K MMM DD    2140 rbd-vc-monitor.log.10
      -rw-r -- r --    1 deploy deploy 976K MMM DD    2140 rbd-watchdog-linux. log.1
      -rw-r -- r --    1 deploy deploy 977K MMM DD    2140 rbd-vc-monitor.log. 9
      -rw-r -- r --    1 deploy deploy 976K MMM DD    2140 rbd-server.log.1
      -rw-r -- r --    1 deploy deploy 977K MMM DD    2140 rbd-vc-monitor.log.8

Note: The above log snippets are for reference only. The actual logs and names respectively may vary.

Cause

There are old traces and corrupted log files under /var/log/vmware/rbd directory of the source vCenter.

Resolution

To resolve this, please follow the steps below:

  1. Take Snapshot of the vCenter (Offline Snapshots in case of Linked Mode vCenters).
  2. Delete the unwanted and old traces of the logs under /var/log/vmware/rbd.
  3. Once completed, restart the vCenter upgrade from Stage 1.

Additional Information

In case Auto Deploy is being used and it's affected after deleting the old/unwanted log traces; please reconfigure the Auto Deploy by referring Preparing for vSphere Auto Deploy

Powered by Wolken Software