vSAN Health Service - Stretched cluster - Witness appliance upgrade to vSphere 7.0 or higher with caution

Products

• VMware vSAN

Issue/Introduction

Symptoms:

Background

Starting with vSphere 7.0, VMKlinux drivers are being deprecated which results in virtual SCSI controllers that use this driver to also be deprecated, such as the LSI Logic Parallel virtual SCSI controller.

Currently, the vSAN virtual Witness Appliance is pre-configured with an LSI Logic Parallel SCSI controller. This means that after upgrading an existing virtual Witness Appliance to 7.0, disks attached to this deprecated controller will not be usable. This health check is to let customer be aware of such issue and provide resolution before performing upgrade.

Why we are removing VMKlinux drivers and what's the impact?

• VMware vSphere Blog - VMware plans to deprecate vmkLinux APIs and associated driver ecosystem
• VMware vSphere Blog - What is the Impact of the VMKlinux Driver Stack Deprecation?

Affected witnesses

• Only virtual Witness Appliances are affected. Physical Witness hoss are not.
• Any virtual Witness Appliance currently deployed that is eligible for a direct 7.0 upgrade is impacted, including direct upgrades to any future releases to or after 7.0.

Environment

VMware vSAN 8.0.x
VMware vSAN 7.0.x

Resolution

Power off the Witness Appliance, reconfigure the virtual SCSI controller to use a Paravirtual SCSI controller and power on before upgrading Witness Appliance to vSphere 7.0

Witness appliance reconfiguration can be done automatically by using the scripts attached to this KB or manually via vCenter Server/ESXi UI. (The controller reconfiguration will not change the SCSi ID order of disk drives).

Witness appliance reconfiguration options:

• Python script
• PowerCLI script
• vCenter Server/ESXi UI

Script

The script is implemented in two languages, Python and PowerShell (VMware PowerCLI). The 2 scripts are zipped and attached to this article.

Python  

Attached script archive: 76948_ChangeScsiController_Python.zip

Synopsis

The script can run on any vCenter or ESXi host which has python installed.

It looks for the Witness in the given datacenter path, powers it off and tries to change the SCSI controller. Once the controller is changed, it powers on the witness and bring it back online.

Note: The script can connect to the vCenter managing the witness or the ESXi host where the witness is currently residing.

Warning: Witness will be powered off when the SCSI controller is being be changed. During that period the components residing on the Witness will be non-compliant

Prerequisites

• Python - versions 2.6.6 and above
• Pyvmomi in the environment or path to the pyvpx folder
• Pip module six
• vCenter or ESXi from where the script executes should be reachable to the witness residing host/vCenter

Sample

python changeScsiController.py -i <vc/host> -u <username> -p <password> -o <port> -n <witness_name> -x <old_controller_type> -y <new_controller_type> -d <datacenter_path> -v <pyvpx_dir_path> -l <log_path>
python changeScsiController.py --help

Inputs to the script: 

'-i' or '–host'	Required field, it is the vCenter managing the witness or the ESX where the witness is currently residing or localhost
'-u' or '–user'	Required field which is the vc/host username
'-p' or '–password'	Required field for logging into the vc/host
'-o' or '–port'	Optional Field which default to 443, it is the hostd port for connection
'-n' or '–name'	Required field which is the name of the witness to be upgraded
'-x' or '–oldCtrl'	Optional Field and defaults to lsilogic, it is the old SCSI controller type and restricted to values - ['buslogic', 'lsilogic', 'lsilogicsas', 'paravirtual']
'-y' or '–newCtrl'	Optional Field and defaults to lsilogic, it is the new SCSI controller type and restricted to values - ['buslogic', 'lsilogic', 'lsilogicsas', 'paravirtual']
-d' or '–datacenter'	Optional Field it is the path to the datacenter where the witness is residing, if provided the witness is searched in that particular datacenter path
'-v' or '–pyvpxdir'	Optional Field, if provided loads the pyvpx from the path to the pyvpx unzipped folder instead of the environment
'-l' or '–logfile'	Optional Field and defaults to /tmp/scsi.log but can be specified to provide log file path for the script logs

 

PowerCLI

Attached script archive: 76948_ChangeScsiController_Powercli.zip

Synopsis

The script can run on any machine(virtual/physical) or ESX host which has PowerCLI installed.

It tries to look for the Witness in the given datacenter path, powers it off and tries to upgrade the SCSI controller.

Once the controller is changed it tries to power on the witness and bring it back online.

Note: The script can connect to the vCenter managing the witness or the ESX where the witness is currently residing.

Warning: Witness will be powered off when the SCSI controller is being be changed. During that period the components residing on the Witness will be non-compliant

Prerequisites

• PowerCLI Version : PowerCLI 11.4.0 and above

• PowerShell Core Version : 6.1 and above

Note: When trying to connect with PowerCLI to a vCenter Server Appliance, you may encounter Invalid server certificate error.

Use Set-PowerCLIConfiguration to set the value for the InvalidCertificateAction option to Prompt if you’d like to connect once or to add a permanent exception for this server.
You may ignore InvalidCertificateAction by running following command before running this script: Set-PowerCLIConfiguration -InvalidCertificateAction Ignore -Confirm:$false

Sample

ChangeScsiController.ps1 usage: ChangeScsiController.ps1 -HostName <hostname> -User <username> -Password <password> -VM <vmname> -Datacenter <datacentername>[optional] -ControllerType <scsi controller type>

Get help output of ChangeScsiController.ps1:
get-help ChangeScsiController.ps1

Inputs to the script:

-HostName	vCenter managing the witness or the ESXi where the witness is currently residing or localhost
-User	User name to use when connecting to host/vCenter
-Password	User Password to use when connecting to host/vCenter
-VM	VM name whose SCSI controller to be changed
-Datacenter	[optional] Datacenter name where vm could be residing
-ControllerType	VM updated with this SCSi controller. Allowed values: [buslogic, lsilogic, lsilogicsas, paravirtual]

 

vCenter Server/ESXi UI

To modify the SCSI controller type in the vSphere Web Client:

1. Login to the vCenter or ESXi UI that Witness Appliance residing

2. Select the Witness Appliance and shut it down 

3. Edit settings of the Witness Appliance, go to "VM Options" and expand "General Options". Make sure "Guest OS Version" is set to "VMware ESXi 6.5 or later". If not, change it. Otherwise "VMware Paravirtual" won't be shown in the next step.

4. Go to "VM Hardware" and expand "SCSI controller 0", select "VMware Paravirtual" for "Change Type". Then click "OK".

5. Power on Witness Appliance

Note: You may observe a warning message after changing controller type to "VMware Paravirtual" which can be ignored.

Alternative method 

Instead of changing the virtual SCSI controller of any existing Witness Appliance to 7.0, re-deployment of a new Witness Appliance in required version later than 7.0 is also possible. New appliances do have the Paravirtual SCSI controller configured.

Option #1 is recommended as this is an alternative workaround.