VCSA upgrade fails at Stage-2 with error "Internal error occurs during import of VMware vSphere auto deploy"
search cancel

VCSA upgrade fails at Stage-2 with error "Internal error occurs during import of VMware vSphere auto deploy"

book

Article ID: 386243

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • During the vCenter major upgrade from vCenter 7.0.x to vCenter 8.0.3, the upgrade process fails at 3rd step of stage-2  "import copied data to target vCenter Server", we may see the error - "A problem has occurred. The source vCenter Server might have been Powered Off during this process".

  • Following error message is observed upon selecting 'Messages' button from the previous error:

Error Resolution: Internal error occurs during Import of VMware vSphere Auto Deploy.

  • Target vCenter appliance's log snippets as follows:
  • /var/log/vmware/upgrade/import-upgrade-runner.log reports 'error occurred while importing the data for autodeploy'.

[YYYY-MM-DDTHH:MM:SS] INFO upgrade.states.component_states rbd:Import: [YYYY-MM-DDTHH:MM:SS]  INFO autodeploy.upgrade_util Ran the registration script stdout:b'', stderr:b'Could not register extension', returncode:42
[YYYY-MM-DDTHH:MM:SS] INFO upgrade.states.component_states rbd:Import: [YYYY-MM-DDTHH:MM:SS]  ERROR autodeploy.upgrade_util Unable to re-register autodeploy service after upgrading
[YYYY-MM-DDTHH:MM:SS] INFO upgrade.states.component_states rbd:Import: [YYYY-MM-DDTHH:MM:SS]  ERROR autodeploy.autodeploy_upgrader Error occurred while importing the data for autodeploy
[YYYY-MM-DDTHH:MM:SS] INFO upgrade.states.component_states rbd:Import: Traceback (most recent call last):
[YYYY-MM-DDTHH:MM:SS] INFO upgrade.states.component_states rbd:Import:     registerAutodeploy(registerScript, xmlPath, vcAddr)
[YYYY-MM-DDTHH:MM:SS] INFO upgrade.states.component_states rbd:Import:   File "/usr/lib/vmware/cis_upgrade_runner/payload/component-scripts/autodeploy/upgrade_util.py", line 154, in registerAutodeploy
[YYYY-MM-DDTHH:MM:SS] INFO upgrade.states.component_states rbd:Import: Exception: Unable to re-register autodeploy service after upgrading
[YYYY-MM-DDTHH:MM:SS] INFO upgrade.states.component_states rbd:Import: Script completed for 16.12973976135254 secs with return-code='1', and executionId=axxxxx-2xxx-4xxx-9xxx-6xxxxxxx
[YYYY-MM-DDTHH:MM:SS] ERROR upgrade.states.component_states rbd:Import: Remote script failed with an error [InternalError()]
[YYYY-MM-DDTHH:MM:SS] ERROR upgrade.states.component_states rbd:Import: failed with internal error. For details take a look at Import_com.vmware.rbd_YYYY_MM_DD_MM_HH.log.
[YYYY-MM-DDTHH:MM:SS] WARNING wf.wf_processor State Import com.vmware.rbd has failed.
[YYYY-MM-DDTHH:MM:SS] ERROR wf.wf_processor Workflow execution has failed
[YYYY-MM-DDTHH:MM:SS] ERROR UpgradeRunner Upgrade mode import failed

  • /var/log/vmware/upgrade/Import_com.vmware.rbd_YYYY_MM_DD_HH_MM.log

[YYYY-MM-DDTHH:MM:SS] INFO autodeploy.upgrade_util The autodeploy register arguments ['/usr/bin/autodeploy-register', '-R', '-a', 'localhost', '-s', '/etc/vmware-rbd/autodeploy-setup.xml', '-l', '-f', '-t', '300']
[YYYY-MM-DDTHH:MM:SS] INFO autodeploy.upgrade_util Ran the registration script stdout:b'', stderr:b'Could not register extension', returncode:42
[YYYY-MM-DDTHH:MM:SS] ERROR autodeploy.upgrade_util Unable to re-register autodeploy service after upgrading
[YYYY-MM-DDTHH:MM:SS] ERROR autodeploy.autodeploy_upgrader Error occurred while importing the data for autodeploy
Traceback (most recent call last):
  File "/usr/lib/vmware/cis_upgrade_runner/payload/component-scripts/autodeploy/autodeploy_upgrader.py", line 316, in importData
    registerAutodeploy(registerScript, xmlPath, vcAddr)
  File "/usr/lib/vmware/cis_upgrade_runner/payload/component-scripts/autodeploy/upgrade_util.py", line 154, in registerAutodeploy
    raise Exception(_error)
Exception: Unable to re-register autodeploy service after upgrading
2025-01-14T11:19:15.745Z ERROR __main__ Upgrade Phase 'autodeploy:Import' failed. Exception: Unable to re-register autodeploy service after upgrading
Traceback (most recent call last):
  File "/usr/lib/vmware/cis_upgrade_runner/payload/componentPhaseLauncher.py", line 461, in main
    executionResult = systemExtension(exeContext)
Exception: Unable to re-register autodeploy service after upgrading

  • /var/log/vmware/rbd/autodeploy-register.log reports no permission to perform the operation.

[YYYY-MM-DDTHH:MM:SS] [44904:MainThread]INFO:vc_extension:Using https://vCenterFQDN:6502 as the managementUrl
[YYYY-MM-DDTHH:MM:SS] [44904:MainThread]INFO:vc_extension:registering servers with full cert and thumbprint -- ##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##
[YYYY-MM-DDTHH:MM:SS] [44904:MainThread]INFO:vc_extension:updating rule extension info
[YYYY-MM-DDTHH:MM:SS] [44904:MainThread]ERROR:autodeploy-register:Could not register extension
Traceback (most recent call last):
  File "/usr/bin/autodeploy-register", line 456, in main
    vc_extension.register(si, configOptions['serviceAddress'],
  File "bora/install/vmvisor/autodeploy/site-packages/vmware/rbd/vc_extension.py", line 456, in register
  File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 618, in <lambda>
    self.f(*(self.args + (obj,) + args), **kwargs)
  File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 391, in _InvokeMethod
    return self._stub.InvokeMethod(self, info, args)
  File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1607, in InvokeMethod
    raise obj  # pylint: disable-msg=E0702
pyVmomi.VmomiSupport.vim.fault.NoPermission: (vim.fault.NoPermission) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   msg = 'Permission to perform this operation was denied.',
   faultCause = <unset>,
   faultMessage = (vmodl.LocalizableMessage) [],
   object = 'vim.Folder:group-d1',
   privilegeId = 'Extension.Update',
   missingPrivileges = (vim.fault.NoPermission.EntityPrivileges) [
      (vim.fault.NoPermission.EntityPrivileges) {
         dynamicType = <unset>,
         dynamicProperty = (vmodl.DynamicProperty) [],
         entity = 'vim.ExtensionManager:ExtensionManager',
         privilegeIds = (str) [
            'Extension.Update'

Environment

  • vCenter 7.x
  • vCenter 8.x

Cause

  • This issue occurs if the vSphere SSO domain (vSphere.local) group 'Administrators' is assigned a role other than the 'Administrator' role defined at global permissions and vCenter level.

Resolution

  • 'Administrators' vSphere SSO domain (vSphere.local) group must have 'Administrator' role assigned to it at global permissions and vCenter level with 'Propagate to children' option enabled. 
    • To validate and update the permission to 'Administrators' vSphere SSO domain group at global permissions:
      • Login to vSphere client as administrator@<vsphere_sso_domain>.
      • Click on '☰' and then select 'Administration'
      • Under 'Access Control' select 'Global Permissions'.
      • The Role assigned to "<vSphere_SSO_Domain>\Administrators" will show Role as anything other than 'Administrator'. 
      • Select the radio button for "<vSphere_SSO_Domain>\Administrators" - User/Group, click on Edit. 
      • On the 'Change Role' window, change the Role to 'Administrator' while keeping the 'Propagate to children' option enabled. Click Ok.
    • To validate and update the permission to 'Administrators' vSphere SSO domain group at global permissions:
      • Login to vSphere client as administrator@<vsphere_sso_domain>.
      • Select vCenter Server name in the inventory. 
      • Select 'Permissions' tab and select the radio button for "<vSphere_SSO_Domain>\Administrators" - User/Group, click on Edit.
      • On the 'Change Role' window, change the Role to 'Administrator' while keeping the 'Propagate to children' option enabled. Click Ok.
Powered by Wolken Software