Error: "Could not authenticate to source appliance OS on address" when attempting vCenter upgrade
search cancel

Error: "Could not authenticate to source appliance OS on address" when attempting vCenter upgrade

book

Article ID: 370215

calendar_today

Updated On: 05-29-2025

Products

VMware vCenter Server

Issue/Introduction

  • When attempting to perform a vCenter upgrade during Stage 2, the "pre-upgrade check" fails with the following error messages: 
    Error: Could not authenticate to source appliance OS on address
Resolution: Verify that source appliance OS credentials are correct
  • Pre-upgrade check failed due the following problem: Error in method invocation [digital envelope routines] unsupported
  • /var/log/vmware/upgrade/upgrade-requirements.log 
    yyyy-mm-ddThh:mm:ss.155Z INFO ssh_utils.ssh_transport Ssh private key /root/.ssh/id_rsa exists, do nothing.
yyyy-mm-ddThh:mm:ss.550Z INFO ssh_utils.ssh_transport Remote ssh authorization file has been updated
yyyy-mm-ddThh:mm:ss.550Z INFO ssh_utils.ssh_transport Ssh password-less transport has been successfully setup
yyyy-mm-ddThh:mm:ss.63Z INFO ssh_utils.ssh_transport ssh command exit code: 255
yyyy-mm-ddThh:mm:ss.64Z WARNING proxy got ('Command ssh completed with exit-code 255. Cause ', 999), retrying ...
yyyy-mm-ddThh:mm:ss.79Z INFO ssh_utils.ssh_transport ssh command exit code: 255
yyyy-mm-ddThh:mm:ss.79Z WARNING proxy got ('Command ssh completed with exit-code 255. Cause ', 999), retrying ...
yyyy-mm-ddThh:mm:ss.92Z INFO ssh_utils.ssh_transport ssh command exit code: 255
yyyy-mm-ddThh:mm:ss.93Z WARNING proxy got ('Command ssh completed with exit-code 255. Cause ', 999), retrying ...
yyyy-mm-ddThh:mm:ss.108Z INFO ssh_utils.ssh_transport ssh command exit code: 255
yyyy-mm-ddThh:mm:ss.108Z WARNING proxy got ('Command ssh completed with exit-code 255. Cause ', 999), retrying ...
yyyy-mm-ddThh:mm:ss.121Z INFO ssh_utils.ssh_transport ssh command exit code: 255
yyyy-mm-ddThh:mm:ss.121Z WARNING proxy got ('Command ssh completed with exit-code 255. Cause ', 999), retries maxed out
yyyy-mm-ddThh:mm:ss.121Z ERROR /usr/lib/vmware/cis_upgrade_runner/libs/sdk/transport/ssh.py Failure while executing command over ssh.
Traceback (most recent call last):
  File "/usr/lib/vmware/cis_upgrade_runner/libs/sdk/transport/ssh.py", line 252, in _executeCommandOverSsh
    return self.sshTransport.executeCommandOverSsh(remoteCommand,
  File "/usr/lib/vmware/cis_upgrade_runner/libs/sdk/proxy.py", line 106, in __call__
    self.ctrl._throwError(e)
  File "/usr/lib/vmware/cis_upgrade_runner/libs/sdk/proxy.py", line 163, in _throwError
    self.errorThrower(e)
  File "/usr/lib/vmware/cis_upgrade_runner/libs/sdk/proxy.py", line 126, in _raise
    raise e
  File "/usr/lib/vmware/cis_upgrade_runner/libs/sdk/proxy.py", line 93, in __call__
    ret = self.func(*args, **kwargs)
  File "/usr/lib/vmware/cis_upgrade_runner/libs/sdk/ssh_utils/ssh_transport.py", line 705, in executeCommandOverSsh
    return self._executeCommand(command, timeout,
  File "/usr/lib/vmware/cis_upgrade_runner/libs/sdk/ssh_utils/ssh_transport.py", line 866, in _executeCommand
    raise ExecutionException('Command %s completed with '
transport.ExecutionException: ('Command ssh completed with exit-code 255. Cause ', 999)
yyyy-mm-ddThh:mm:ss.122Z ERROR upgrade_commands Could not authenticate to source appliance OS on address vcenter.example.com
Traceback (most recent call last):
  File "/usr/lib/vmware/cis_upgrade_runner/libs/sdk/transport/ssh.py", line 252, in _executeCommandOverSsh
    return self.sshTransport.executeCommandOverSsh(remoteCommand,
  File "/usr/lib/vmware/cis_upgrade_runner/libs/sdk/proxy.py", line 106, in __call__
    self.ctrl._throwError(e)
  File "/usr/lib/vmware/cis_upgrade_runner/libs/sdk/proxy.py", line 163, in _throwError
    self.errorThrower(e)
  File "/usr/lib/vmware/cis_upgrade_runner/libs/sdk/proxy.py", line 126, in _raise
    raise e
  File "/usr/lib/vmware/cis_upgrade_runner/libs/sdk/proxy.py", line 93, in __call__
    ret = self.func(*args, **kwargs)
  File "/usr/lib/vmware/cis_upgrade_runner/libs/sdk/ssh_utils/ssh_transport.py", line 705, in executeCommandOverSsh
    return self._executeCommand(command, timeout,
  File "/usr/lib/vmware/cis_upgrade_runner/libs/sdk/ssh_utils/ssh_transport.py", line 866, in _executeCommand
    raise ExecutionException('Command %s completed with '
transport.ExecutionException: ('Command ssh completed with exit-code 255. Cause ', 999)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/vmware/cis_upgrade_runner/bootstrap_scripts/upgrade_commands.py", line 3962, in _getLinuxCommandFactory
    opsManager = createSshOperationManager(
  File "/usr/lib/vmware/cis_upgrade_runner/libs/sdk/transport/ssh.py", line 107, in createSshOperationManager
    return SshOperationManager(vmAddress, guestOps.vmIp, guestUser,
  File "/usr/lib/vmware/cis_upgrade_runner/libs/sdk/transport/ssh.py", line 213, in __init__
    self.setup()
  File "/usr/lib/vmware/cis_upgrade_runner/libs/sdk/transport/ssh.py", line 231, in setup
    self._getWorkingDirectory()
  File "/usr/lib/vmware/cis_upgrade_runner/libs/sdk/transport/ssh.py", line 297, in _getWorkingDirectory
    self.workingDirectory = self._executeCommandOverSsh(command).stdout.strip()
  File "/usr/lib/vmware/cis_upgrade_runner/libs/sdk/transport/ssh.py", line 256, in _executeCommandOverSsh
    raise FileException('Cannot execute over ssh as retries have been exhausted')
transport.FileException: Cannot execute over ssh as retries have been exhausted
yyyy-mm-ddThh:mm:ss.123Z ERROR root Could not create command factory.
yyyy-mm-ddThh:mm:ss.123Z ERROR __main__ ERROR: Fatal error during upgrade REQUIREMENTS. For more details take a look at: /var/log/vmware/upgrade/requirements-upgrade-runner.log
yyyy-mm-ddThh:mm:ss.124Z INFO root Exiting with exit-code 1

If the upgrade was triggered via the SDDC manager then the logs from the sddc will look like the following: 

/var/log/vmware/vcf/lcm/vcenter/875cb166-6315-4d9e-af66-c61d3eb6de79/5b50bf8e-a7d1-4234-a515-b8cc2171c571/install/workflow_1736928168469/input

taskflow.log:

2025-01-15 08:11:53,737 [loggable.py:102]: VCSA Deployment is still running
2025-01-15 08:12:23,914 [loggable.py:102]: Retrieved all ip addresses from appliance vcenter-upgrade-target-appliance_d7d30979-438c-42b8-bb0a-183b2c093d61: ['10.xx.xx.xx']
2025-01-15 08:12:23,927 [loggable.py:102]: Querying REST endpoint '/rest/vcenter/deployment' on appliance '10.xx.xx.xx' for deployment status
2025-01-15 08:12:24,044 [taskflow.py:1038]: <MonitorVCDeploymentTask - com.vmware.vcsa.installer.monitor.monitor_vcsa_deployment(RUNNING)> in <UpgradeTaskFlow - input(RUNNING)> status changed to: RUNNING
2025-01-15 08:12:24,154 [taskflow.py:1038]: <MonitorVCDeploymentTask - com.vmware.vcsa.installer.monitor.monitor_vcsa_deployment(RUNNING)> in <UpgradeTaskFlow - input(RUNNING)> status changed to: RUNNING
2025-01-15 08:12:24,206 [loggable.py:102]: ==========VCSA Deployment Progress Report==========
        Task: Install required RPMs for the appliance.(SUCCEEDED 100/100)       - Task has completed successfully.
        Task: Run appliance precheck.(FAILED 0/100)     -
                Error: Could not authenticate to source appliance OS on address vcsa_fqdn.
                Resolution: Verify that source appliance OS credentials are correct.

2025-01-15 08:12:24,206 [loggable.py:102]: The VCSA deployment has failed
VCSA Deployment Start Time: 2025-01-15T08:07:55.771Z
VCSA Deployment End Time: 1970-01-01T00:00:00.000Z

2025-01-15 08:12:24,207 [loggable.py:102]: Ready to collect support bundle from deployed appliance, if applicable
2025-01-15 08:12:24,355 [common_func.py:223]: Appliance 'vcenter-upgrade-target-appliance_d7d30979-438c-42b8-bb0a-183b2c093d61' is ready for guest operations
2025-01-15 08:12:24,356 [common_func.py:226]: Collecting the support bundle from the deployed appliance 'vcenter-upgrade-target-appliance_d7d30979-438c-42b8-bb0a-183b2c093d61' on ESXi 'ESXi_FQDN'
2025-01-15 08:13:28,862 [loggable.py:102]: Successfully collected support bundle and stored at: /var/log/vmware/vcf/lcm/vcenter/875cb166-6315-4d9e-af66-c61d3eb6de79/5b50bf8e-a7d1-4234-a515-b8cc2171c571/install/workflow_1736928168469/input/vc-support-bundle.tgz
2025-01-15 08:13:28,863 [loggable.py:102]: The support bundle /var/log/vmware/vcf/lcm/vcenter/875cb166-6315-4d9e-af66-c61d3eb6de79/5b50bf8e-a7d1-4234-a515-b8cc2171c571/install/workflow_1736928168469/input/vc-support-bundle.tgz is not a valid tarfile, it might have been broken during download
2025-01-15 08:13:28,863 [loggable.py:102]: Reverting the IP address of the appliance 'vcenter-upgrade-target-appliance_d7d30979-438c-42b8-bb0a-183b2c093d61'.
2025-01-15 08:13:30,149 [loggable.py:102]: Failed to revert the IP address of the appliance 'vcenter-upgrade-target-appliance_d7d30979-438c-42b8-bb0a-183b2c093d61'. Exception: Cannot run the revert networking command. revert_networking.py doesn't exist on target VC. Exception: Failed to run and wait for command in guest with error 'Command '['ls', '`install-parameter upgrade.import.directory`/system-data/revert_networking.py']' exited with non-zero status 2' with error 'Cannot run the revert networking command. revert_networking.py doesn't exist on target VC. Exception: Failed to run and wait for command in guest with error 'Command '['ls', '`install-parameter upgrade.import.directory`/system-data/revert_networking.py']' exited with non-zero status 2''
2025-01-15 08:13:30,165 [loggable.py:102]: ============== [FAILED] Task: MonitorDeploymentTask: Monitoring Deployment execution failed at 08:13:30 AM ==============
2025-01-15 08:13:30,165 [loggable.py:102]: Task 'MonitorDeploymentTask: Monitoring Deployment' execution failed because [The VCSA deployment has failed

 

Note: the IP address from the line  "2025-01-15 08:12:23,927 [loggable.py:102]: Querying REST endpoint '/rest/vcenter/deployment' on appliance '10.xx.xx.xx" refers to the temp IP and the API is queried to the target VC with the temp IP. 

Environment

VMware vSphere ESXi 7.x
VMware vCenter Server 7.x
VMware vSphere ESXi 8.x
VMware vCenter Server 8.x

VCF 4.x VCF 5.x 

Cause

The sshd_config file has been malformed or modified for security enhancement. 

 

Resolution

To resolve this issue:

  1. SSH to the source vCenter
  2. Change directory to /etc/ssh: 
    cd /etc/ssh/
  3. Back up the existing ssh_config file: 
    cp sshd_config sshd_config.backup

  4. Replace sshd_config with an unedited version (i.e.: from another vCenter)
  5. Re-attempt the upgrade

Note: If step 4 cannot be completed because there is only one vCenter in the environment, contact Broadcom support for assistance 

Powered by Wolken Software