Upgrading vCenter from 7.x to 8.x fails with error "Pre-upgrade check failed due the following problem: Error in method invocation [digital envelope routines] unsupported"

Products

• VMware vCenter Server
• VMware vCenter Server 8.0
• VMware vCenter Server 7.0

Issue/Introduction

• When attempting to perform a vCenter upgrade during Stage 2, the "pre-upgrade check" fails with the following error messages: Pre-upgrade check failed due the following problem: Error in method invocation [digital envelope routines] unsupported
  
  
• Logs on the source vCenter Server /var/log/vmware/upgrade/upgrade-requirements.log will show similar to below snippets:

yyyy-mm-ddThh:mm:ss INFO os_utils Trying to retrieve the host <<VC-IP>> python3 path
yyyy-mm-ddThh:mm:ss ERROR transport Command ['/usr/bin/python3', '/tmp/vmware-upgrade-temp-dirHOHQAASRn0/tmpCrRDKAbupG/bootstrap_scripts/run_linux_preupgrade_checks.py', '--inputFile', 'upgrade-requirements-config.json', '--outputFile', 'upgrade-requirements-output.json', '--logDir', '/var/log/vmware/upgr
ade', '--logFileName', 'upgrade-source-requirements.log', '--disableScreenLog', '--logLevel', 'INFO'] exit-code=1, stdout=, stderr=Caught unhandled exception. Error: Traceback (most recent call last):
  File "/tmp/vmware-upgrade-temp-dirHOHQAASRn0/tmpCrRDKAbupG/bootstrap_scripts/run_linux_preupgrade_checks.py", line 1682, in <module>
    sys.exit(main(sys.argv))
  File "/tmp/vmware-upgrade-temp-dirHOHQAASRn0/tmpCrRDKAbupG/bootstrap_scripts/run_linux_preupgrade_checks.py", line 1553, in main
    level=execData['logLevel'], disableScreenLog=execData['disableScreenLog'])
  File "/tmp/vmware-upgrade-temp-dirHOHQAASRn0/tmpCrRDKAbupG/bootstrap_scripts/common.py", line 115, in setupLogging
    createDirectory(logDir)
  File "/tmp/vmware-upgrade-temp-dirHOHQAASRn0/tmpCrRDKAbupG/bootstrap_scripts/common.py", line 89, in createDirectory
    os.makedirs(folder)
  File "/usr/lib/python3.7/os.py", line 221, in makedirs
    mkdir(name, mode)
OSError: [Errno 28] No space left on device: '/var/log/vmware/upgrade'
yyyy-mm-ddThh:mm:ss ERROR upgrade_commands Pre-upgrade checks failed. Check upgrade-source-requirements.log log for details.
yyyy-mm-ddThh:mm:ss ERROR upgrade_commands Unable to download pre-upgrade checks result file /tmp/vmware-upgrade-temp-dirHOHQAASRn0/tmpCrRDKAbupG/upgrade-requirements-output.json from <<VC-IP>>
yyyy-mm-ddThh:mm:ss INFO upgrade_commands Reporting source preupgrade result errors and warnings.
yyyy-mm-ddThh:mm:ss WARNING upgrade_commands Pre-upgrade checks warnings:
yyyy-mm-ddThh:mm:ss WARNING upgrade_commands {'text': {'id': 'ur.esx.managedby.vc.warning.text', 'translatable': 'This ESXi host [%(0)s] is managed by vCenter Server [%(1)s].', 'args': ['//<<ESXI>>:443', '<<VC-IP>>'], 'localized': 'This ESXi host [//<<ESXI>>:443] is managed by vCenter Server [<<VC-IP>>].'}, 'description': None, 'resolution': {'id': 'ur.esx.managedby.source.vc.resolution', 'translatable': 'Make sure the cluster where this ESXi host resides is not set to Fully Automated DRS for the duration of the upgrade process.', 'localized': 'Make sure the cluster where this ESXi host resides is not set to Fully Automated DRS for the duration of the upgrade process.'}, 'problemId': None}
yyyy-mm-ddThh:mm:ss:37.810Z ERROR upgrade_commands Pre-upgrade checks errors:
yyyy-mm-ddThh:mm:ss:37.811Z ERROR upgrade_commands {'text': {'id': 'ur.internal.text', 'translatable': 'Internal error occurs during execution of upgrade process.', 'localized': 'Internal error occurs during execution of upgrade process.'}, 'description': None, 'resolution': {'id': 'ur.internal.resolution', 'translatable': 'Please search for these symptoms in the VMware Knowledge Base for any known issues and possible resolutions. If none can be found, collect a support bundle and open a support request.', 'localized': 'Please search for these symptoms in the VMware Knowledge Base for any known issues and possible resolutions. If none can be found, collect a support bundle and open a support request.'}, 'problemId': None}
yyyy-mm-ddThh:mm:ss ERROR upgrade_commands Pre-upgrade checks failed on source host. Check upgrade-source-requirements.log log for  details.
yyyy-mm-ddThh:mm:ss ERROR upgrade_commands Skipping rest of target checks as no preupgrade result is provided
yyyy-mm-ddThh:mm:ss INFO upgrade_commands Prune InstallInfo spec
yyyy-mm-ddThh:mm:ss INFO upgrade_commands Persisting preupgrade-checks to "/var/log/vmware/upgrade/prechecks.json"
yyyy-mm-ddThh:mm:ss INFO upgrade_commands Persisting preupgrade-checks to temporary file "/var/log/vmware/upgrade/prechecks.json.tmp"
yyyy-mm-ddThh:mm:ss INFO base_commands Cleaning up remote host...
yyyy-mm-ddThh:mm:ss INFO base_commands Download directory /var/log/vmware/upgrade to /var/log/vmware/upgrade
yyyy-mm-ddThh:mm:ss INFO ssh_utils.ssh_transport ssh command exit code: 1
yyyy-mm-ddThh:mm:ss ERROR base_commands Unable to upload required files to [<<VC-IP>>]
Traceback (most recent call last):
  File "/usr/lib/vmware/cis_upgrade_runner/bootstrap_scripts/base_commands.py", line 516, in execute
    self.opsManager.downloadDirectory(destinationPath, sourcePath)
  File "/usr/lib/vmware/cis_upgrade_runner/libs/sdk/transport/ssh.py", line 151, in func_wrapper
    return func(self, *args, **kwargs)
  File "/usr/lib/vmware/cis_upgrade_runner/libs/sdk/transport/ssh.py", line 633, in downloadDirectory
    self.sshTransport.downloadDirectory(targetPath,
  File "/usr/lib/vmware/cis_upgrade_runner/libs/sdk/proxy.py", line 98, in __call__
    self.ctrl._throwError(e)
  File "/usr/lib/vmware/cis_upgrade_runner/libs/sdk/proxy.py", line 163, in _throwError
    self.errorThrower(e)
  File "/usr/lib/vmware/cis_upgrade_runner/libs/sdk/proxy.py", line 126, in _raise
    raise e
  File "/usr/lib/vmware/cis_upgrade_runner/libs/sdk/proxy.py", line 93, in __call__
    ret = self.func(*args, **kwargs)
  File "/usr/lib/vmware/cis_upgrade_runner/libs/sdk/ssh_utils/ssh_transport.py", line 917, in downloadDirectory
    raise FileException('Cannot download directory because remote '
transport.FileException: Cannot download directory because remote directory /var/log/vmware/upgrade does not exist
yyyy-mm-ddThh:mm:ss INFO base_commands Executing command: [<class 'base_commands.RemoveFilesCommand'>]
yyyy-mm-ddThh:mm:ss INFO ssh_utils.ssh_transport ssh command exit code: 0
yyyy-mm-ddThh:mm:ss INFO base_commands Directory /tmp/vmware-upgrade-temp-dirHOHQAASRn0/tmpCrRDKAbupG is successfully removed
yyyy-mm-ddThh:mm:ss INFO base_commands Executing command: [<class 'upgrade_commands.CleanupRemoteLinuxTransportCommand'>]
yyyy-mm-ddThh:mm:ss INFO upgrade_commands Cleanup SSH transport on remote host: "<<VC-IP>>"
yyyy-mm-ddThh:mm:ss INFO ssh_utils.ssh_transport ssh command exit code: 0
yyyy-mm-ddThh:mm:ss INFO featureState_utils Found FSS 'Ssh_Management_Phase1' with value 'False'
yyyy-mm-ddThh:mm:ss INFO ssh_utils.ssh_transport ssh stderr: mkdir: cannot create directory <91>/var/log/vmware/upgrade<92>: No space left on device
yyyy-mm-ddThh:mm:ss INFO ssh_utils.ssh_transport ssh command exit code: 1

 

Environment

VMware vCenter Server 7.x
VMware vCenter Server 8.x

Cause

Potential causes include:

• vCenter Server log bundles not being cleared after generation.
• Very high frequency events filling up logs.

Resolution

To investigate the issue of vCenter Server /storage/log showing as full, follow these steps:

1. SSH into the vCenter Server appliance.

2. Navigate to the /storage/log directory:

# cd /storage/log 

3. Identify and clean-up old vCenter log bundles if not required. 

4. Run the following command to identify the largest files:

# find . -type f -print0 | xargs -0 du -h | sort -rh | head -n 10

5. Based on the output, refer the KB: vCenter log disk exhaustion or /storage/log full