Pre-Upgrade Check Failure: Method Invocation Error – Unsupported Digital Envelope Routine
search cancel

Pre-Upgrade Check Failure: Method Invocation Error – Unsupported Digital Envelope Routine

book

Article ID: 388499

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

vCenter Server upgrade from version 7.x to 8.x fails during the requirements check phase.

On the target vCenter (8.x), check for the log entries below located under: /var/log/vmware/upgrade/upgrade-requirements.log

YYYY-MM-DDTHH:MM:SS.Z INFO ssh_utils.ssh_transport Ssh password-less transport has been successfully setup
YYYY-MM-DDTHH:MM:SS.Z INFO ssh_utils.ssh_transport ssh command exit code: 255
YYYY-MM-DDTHH:MM:SS.Z WARNING proxy got ('Command ssh completed with exit-code 255. Cause ', 999), retrying ...
:
:
YYYY-MM-DDTHH:MM:SS.Z WARNING proxy got ('Command ssh completed with exit-code 255. Cause ', 999), retries maxed out
YYYY-MM-DDTHH:MM:SS.Z ERROR /usr/lib/vmware/cis_upgrade_runner/libs/sdk/transport/ssh.py Failure while executing command over ssh.
Traceback (most recent call last):
  File "/usr/lib/vmware/cis_upgrade_runner/libs/sdk/transport/ssh.py", line 252, in _executeCommandOverSsh
    return self.sshTransport.executeCommandOverSsh(remoteCommand,
  File "/usr/lib/vmware/cis_upgrade_runner/libs/sdk/proxy.py", line 106, in __call__

Traceback (most recent call last):
  File "/usr/lib/vmware/cis_upgrade_runner/bootstrap_scripts/upgrade_commands.py", line 3968, in _getLinuxCommandFactory
    opsManager = createSshOperationManager(
  File "/usr/lib/vmware/cis_upgrade_runner/libs/sdk/transport/ssh.py", line 107, in createSshOperationManager
    return SshOperationManager(vmAddress, guestOps.vmIp, guestUser,
  File "/usr/lib/vmware/cis_upgrade_runner/libs/sdk/transport/ssh.py", line 213, in __init__
    self.setup()
  File "/usr/lib/vmware/cis_upgrade_runner/libs/sdk/transport/ssh.py", line 231, in setup
    self._getWorkingDirectory()
  File "/usr/lib/vmware/cis_upgrade_runner/libs/sdk/transport/ssh.py", line 297, in _getWorkingDirectory
    self.workingDirectory = self._executeCommandOverSsh(command).stdout.strip()
  File "/usr/lib/vmware/cis_upgrade_runner/libs/sdk/transport/ssh.py", line 256, in _executeCommandOverSsh
    raise FileException('Cannot execute over ssh as retries have been exhausted')
transport.FileException: Cannot execute over ssh as retries have been exhausted
YYYY-MM-DDTHH:MM:SS.Z ERROR root Could not create command factory.
YYYY-MM-DDTHH:MM:SS.Z ERROR __main__ ERROR: Fatal error during upgrade REQUIREMENTS. For more details take a look at: /var/log/vmware/upgrade/requirements-upgrade-runner.log

 

Environment

  • vCenter Server 7.x
  • vCenter Server 8.x

Cause

During a vCenter upgrade, the destination vCenter establishes an SSH connection to the source vCenter.
For this connection to succeed, the destination vCenter’s SSH key must be added to the authorized_keys file of the source vCenter (typically located at /root/.ssh/authorized_keys).

In this case, the source vCenter was reverted to a snapshot taken after Stage 1 of the upgrade, which resulted in an outdated authorized_keys file. This file no longer contains the public key of the destination vCenter, causing authentication to fail.

Example:

To verify, open an SSH session to the source vCenter and run:

ls -lah /root/.ssh/authorized_keys

Output

-rw------- 1 root root 0 Dec  2  2021 /root/.ssh/authorized_keys

The date indicates that the file has not been updated recently in source under authorized_keys. If the destination vCenter's key had been stored successfully, this file should reflect a recent timestamp. 

Resolution

Step 1: Connect to the source appliance through SSH

1. Login to Source(7.0) vCenter server using SSH with root credentials.
2. Switch the vCenter server appliance to the Bash shell using command .Refer to KB : Toggling the vCenter Server Appliance default shell

cat /etc/ssh/sshd_config | grep PubkeyAuthentication


3. Validate the SSH configuration file using the below command and  Ensure the following line is uncommented: 'PubkeyAuthentication yes' . If commented use a text editor to uncomment this line.

cat /etc/ssh/sshd_config | grep PubkeyAuthentication

     
    
4. Restart the SSH service to apply the changes if any:

systemctl restart sshd


Step 2: Enable Passwordless SSH Authentication

1. Login to vCenter 8.0 appliance (Destination) using SSH with root credentials .
2. Run the below command to generate SSH Key Pair:

ssh-keygen -t rsa -b 2048

Note:
Enter to accept the default file location: /root/.ssh/id_rsa. Leave the passphrase empty and press enter.
This will generate two files under /root/.ssh:

id_rsa
id_rsa.pub

3. Copy Public Key (id_rsa.pub) to Source vCenter (7.0) under /tmp directory
4. SSH into the vCenter 7.0 appliance and run:

mkdir -p ~/.ssh
cat /tmp/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

5. Verify Passwordless SSH from vCenter 8.0 to 7.0 using the below command:

ssh root@<7.0-appliance-IP>

Note: It should not prompt for a password.

6. Repeat the Above Steps in from vCenter 7.0 to vCenter 8.0 using the same process.
7. Restart all services on both vCenter appliances :

service-control --stop --all 
service-control --start --all


8. Confirm Passwordless SSH is working before initiating the upgrade.
9. Proceed with the Upgrade




Additional Information

Error: "Could not authenticate to source appliance OS on address" when attempting vCenter upgrade

To edit a file using vi editor, refer KB : Adding/Deleting/Editing a host entry on vCenter server or ESXi host using vi editor

If the /root/.ssh directory on the source vCenter has incorrect permissions or ownership, it can prevent the destination vCenter from establishing an SSH connection during Stage 2 of the vCenter upgrade process. Verify that the "/root/.ssh" directory is set for "drwx------". If it is not, you may observe the following errors in the journalctl logs on the source vCenter:

YYYY-MM-DDTHH:MM:SS.Z sshd[45497]: Authentication refused: bad ownership or modes for directory /root/.ssh
YYYY-MM-DDTHH:MM:SS.Z sshd[45827]: Authentication refused: bad ownership or modes for directory /root/.ssh

Run the following command to set the proper permissions on the SSH directory:

chmod 700 ~/.ssh

Ensuring correct permissions allows the SSH authentication to complete successfully, enabling the upgrade process to proceed.

Powered by Wolken Software