"Encountered an Internal Error" While Upgrading vCenter Server Appliance (VCSA) to a Newer Version

Products

• VMware vCenter Server

Issue/Introduction

When upgrading the vCenter Server Appliance (VCSA) to a newer version, the upgrade fails with the following error message:

Encountered an internal error.
Traceback (most recent call last):
  File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 2018, in main
    vmidentityFB.boot()
  File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 349, in boot
    self.configureSTS(self.__stsRetryCount, self.__stsRetryInterval)
  File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 1479, in configureSTS
    self.startSTSService()
  File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 1141, in startSTSService
    returnCode = self.startService(self.__sts_service_name, self.__stsRetryCount * self.__stsRetryInterval)
  File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 88, in startService
    return service_start(svc_name, wait_time)
  File "/usr/lib/vmware/site-packages/cis/utils.py", line 784, in service_start
    raise ServiceStartException(svc_name)

ServiceStartException: {
  "resolution": null,
  "detail": [
    {
      "args": [ "vmware-stsd" ],
      "id": "install.ciscommon.service.failstart",
      "localized": "An error occurred while starting service 'vmware-stsd'",
      "translatable": "An error occurred while starting service '%(0)s'"
    }
  ],
  "componentKey": null,
  "problemId": null
}

 

In the /var/log/firstboot/vmidentity-firstboot.py_stderr.log file, you may see log entries similar to:

YYYY-MM-DD HH:MM:SS Failure setting accounting for vmware-sts-idmd. Err Failed to set unit properties on vmware-sts-idmd.service: Unit vmware-sts-idmd.service is not loaded.
YYYY-MM-DD HH:MM:SS Failure setting accounting for vmware-stsd. Err Failed to set unit properties on vmware-stsd.service: Unit vmware-stsd.service is not loaded.
YYYY-MM-DD HH:MM:SS Invoked command: ['/sbin/service', 'vmware-stsd', 'start']
YYYY-MM-DD HH:MM:SS RC = 1 Stdout = Stderr = Job for vmware-stsd.service failed because a timeout was exceeded. See "systemctl status vmware-stsd.service" and "journalctl -xe" for details.
YYYY-MM-DD HH:MM:SS {
  "resolution": null,
  "detail": [
    {
      "args": [
        "Command: ['/sbin/service', 'vmware-stsd', 'start']\nStderr: Job for vmware-stsd.service failed because a timeout was exceeded. See \"systemctl status vmware-stsd.service\" and \"journalctl -xe\" for details.\n"
      ],
      "id": "install.ciscommon.command.errinvoke",
      "localized": "An error occurred while invoking external command : 'Command: ['/sbin/service', 'vmware-stsd', 'start']\nStderr: Job for vmware-stsd.service failed because a timeout was exceeded. See \"systemctl status vmware-stsd.service\" and \"journalctl -xe\" for details.\n'",
      "translatable": "An error occurred while invoking external command : '%(0)s'"
    }
  ],
  "componentKey": null,
  "problemId": null
}

 

Additionally, running the cat /etc/resolv.conf command may return DNS entries that are no longer valid or not in use.

Environment

• VMware vCenter Server Appliance 6.5.x

• VMware vCenter Server Appliance 7.0.

Cause

This issue can occur if the vCenter Server Appliance is configured to use an incorrect or outdated DNS server. The misconfiguration prevents required services, such as vmware-stsd, from starting successfully during the upgrade process.

Resolution

To resolve this issue:

1. Revert to the original (source) Appliance if the upgrade has failed and is unrecoverable.

2. Correct the DNS settings via the vSphere Client:

   • Log in to the vSphere Client.

   • Navigate to the Networking settings of the source Appliance.

   • Update the DNS entries to reflect the correct, reachable DNS servers.

3. Retry the upgrade process.

Note: If the issue persists after correcting DNS and retrying the upgrade, collect a support bundle and open a Support Request with VMware by Broadcom for further assistance.

Additional Information

 

• DNS resolution is critical during firstboot, especially for services that require service token signing (STS).