In-place upgrade of the guest operating system (between major OS release versions, such as Windows 10, version 2004 or later to Windows 11) is not recommended. The best practice is to install a new major OS releases in a new VM.
Before guest OS upgrade, upgrading VMware tools to the latest version available for current Windows release, test installation on a cloned VM firstly, taking snapshot of the VM or saving backup of the VM are recommended.
In Windows 10 guest OS, download and run PC Health Check app (https://www.microsoft.com/windows/windows-11#pchealthcheck) to check if your virtual machine meets the minimum system requirements.
Change virtual machine virtual hardware configurations according to the result of PC Health Check when needed. Note: do not change the firmware from BIOS to EFI, or the guest OS will not boot up. For adding vTPM device to an existing virtual machine, please refer to this VMware doc: https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.security.doc/GUID-4DBF65A4-4BA0-4667-9725-AE9F047DE00A.html
If you have a Windows 11 ISO image used for Windows 11 installation, follow the resolution given below,
Mount this ISO file to the VM, select “Virtual Machine -> Edit Settings -> CD/DVD drive -> Datastore ISO file”, click “Browse” to select the Windows 11 ISO file from datastore as CD/DVD media.
Logon to the guest OS and double-click CD-ROM from the File Explorer.
Follow the installation prompts to complete the installation.
Upgrade or reinstall VMware tools after OS upgrade is done.
Shutdown VM, select “Virtual Machine -> Edit Settings -> VM Options -> General Options”, change “Guest OS Version” to “Microsoft Windows 11 (64-bit)” on vSphere 8.0. For Windows 11 guest operating system option see this KB85665 .
For other methods please refer to Microsoft doc: Ways to install Windows 11
Known issue:
Guest OS in-place upgrade to Windows Server 2022 or Windows 11 failed after upgrading VMware Tools to 12.0.5 or later on vSphere (https://kb.vmware.com/s/article/89290)
Support considerations: