After upgrade to ESXi 7.0 U1, vsanperf daemon no longer starts citing 'No module named 'urllib3' During handling of the above exception'

Products

• VMware vSAN
• VMware vSphere ESXi

Issue/Introduction

Symptoms:
After host update, the host will no longer function or be usable by vsan.

Following alerts will be seen:
Host is in a vSAN cluster but does not have vSAN service enabled
vSAN network is not configured
vSphere HA agent on this host is disabled

vsanmgmt log will show the following: 
 

Error: No module named 'urllib3' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/build/mts/release/bora-17325551/bora/build/esx/release/vsanhealth/usr/lib/vmware/vsan/perfsvc/vsanperfsvc.py", line 178, in main File "/build/mts/release/bora-

17325551/bora/build/esx/release/vsanhealth/usr/lib/vmware/vsan/perfsvc/statsdaemon.py", line 72, in <module> SystemExit: 1

Cause

This issue is due to a change in best practices when it comes to upgrading ESXi hosts via command line. When running the update via the 'esxclli software vib install' command, the install will skip over installing essential vibs and therefore will leave the host in an unworkable state. 

Resolution

Instead of using 'esxcli software vib install' to upgrade hosts after 7.0 and onward, it is instead recommended to use 'esxcli software profile update' 

Please see:
https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.esxi.upgrade.doc/GUID-1E773360-CB1C-4BC2-B2A4-B73AB5706FAF.html

Our documentation was updated in 7.0 with this support statement:
"The esxcli software vib update and esxcli software vib install commands are not supported for upgrade operations. See Upgrade or Update a Host with Image Profiles."


Resolution:

After uploading the Zip file into a datastore accessible by the host

• Run the following command: esxcli software sources profile list -d <path to zip file>

This will list the available profiles to install in that zip.

example:
esxcli software sources profile list -d  /vmfs/volumes/5bbcc5e4-6a5a9540-0225-246e96ca8f30/vmkdump/VMware-VMvisor-Installer-7.0.0.update01-17325551.x86_64-DellEMC_Customized-A01.zip
246e96ca8f30/vmkdump/VMware-VMvisor-Installer-7.0.0.update01-17325551.x86_64-DellEMC_Customized-A01.zip
Name                       Vendor     Acceptance Level  Creation Time        Modification Time
-------------------------  ---------  ----------------  -------------------  -----------------
DEL-ESXi-701_17325551-A01  Dell Inc.  PartnerSupported  2020-12-18T21:03:56  2020-12-18T21:03:56
-----
In this example the profile is 'DEL-ESXI-701_1732551-A01'
-----

• Next run the following command to install the update: esxcli software profile update -p <insert profile here> -d <insert file path here>

Example: (Using above information still)
esxcli software profile update -p DEL-ESXi-701_17325551-A01 -d /vmfs/volumes/5bbcc5e4-6a5a9540-0225-246e96ca8f30/vmkdump/VMware-VMvisor-Installer-7.0.0.update01-17325551.x86_64-DellEMC

_Customized-A01.zip
Update Result
   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
   Reboot Required: true
   VIBs Installed: VMW_bootbank_brcmfcoe_12.0.1500.0-1vmw.701.0.0.16850804, VMW_bootbank_brcmnvmefc_12.6.278.10-3vmw.701.0.0.16850804, VMW_bootbank_elxiscsi_12.0.1200.0-2vmw.701.0.0.16850804, VMW_bootbank_elxnet_12.0.1250.0-5vmw.701.0.0.16850804, VMW_bootbank_i40iwn_1.1.2.6-1vmw.701.0.0.16850804, VMW_bootbank_iavmd_2.0.0.1055-5vmw.701.0.0.16850804, VMW_bootbank_icen_1.0.0.9-1vmw.701.0.0.16850804, VMW_bootbank_iser_1.1.0.1-1vmw.701.0.0.16850804, VMW_bootbank_lpfc_12.6.278.10-8vmw.701.0.0.16850804, VMW_bootbank_lpnic_11.4.62.0-1vmw.701.0.0.16850804, VMW_bootbank_lsi-msgpt2_20.00.06.00-3vmw.701.0.0.16850804, VMW_bootbank_lsi-msgpt3_17.00.10.00-2vmw.701.0.0.16850804, VMW_bootbank_mtip32xx-native_3.9.8-1vmw.701.0.0.16850804, VMW_bootbank_ne1000_0.8.4-11vmw.701.0.0.16850804, VMW_bootbank_nenic_1.0.29.0-2vmw.701.0.0.16850804, VMW_bootbank_nfnic_4.0.0.44-2vmw.701.0.0.16850804, VMW_bootbank_nhpsa_70.0051.0.100-2vmw.701.0.25.17325551, VMW_bootbank_nmlx4-core_3.19.16.8-2vmw.701.0.0.16850804, VMW_bootbank_nmlx4-en_3.19.16.8-2vmw.701.0.0.16850804, VMW_bootbank_nmlx4-rdma_3.19.16.8-2vmw.701.0.0.16850804, VMW_bootbank_ntg3_4.1.5.0-0vmw.701.0.0.16850804, VMW_bootbank_nvme-pcie_1.2.3.9-2vmw.701.0.0.16850804, VMW_bootbank_nvmerdma_1.0.1.2-1vmw.701.0.20.17325020, VMW_bootbank_nvmxnet3-ens_2.0.0.22-1vmw.701.0.0.16850804, VMW_bootbank_nvmxnet3_2.0.0.30-1vmw.701.0.0.16850804, VMW_bootbank_pvscsi_0.1-2vmw.701.0.0.16850804, VMW_bootbank_qflge_1.1.0.11-1vmw.701.0.0.16850804, VMW_bootbank_rste_2.0.2.0088-7vmw.701.0.0.16850804, VMW_bootbank_sfvmk_2.4.0.0010-15vmw.701.0.0.16850804, VMW_bootbank_smartpqi_70.4000.0.100-4vmw.701.0.25.17325551, VMW_bootbank_vmkata_0.1-1vmw.701.0.25.17325551, VMW_bootbank_vmkfcoe_1.0.0.2-1vmw.701.0.20.17325020, VMW_bootbank_vmkusb_0.1-1vmw.701.0.25.17325551, VMW_bootbank_vmw-ahci_2.0.5-2vmw.701.0.0.16850804, VMware_bootbank_cpu-microcode_7.0.1-0.25.17325551, VMware_bootbank_crx_7.0.1-0.25.17325551, VMware_bootbank_elx-esx-libelxima.so_12.0.1200.0-3vmw.701.0.0.16850804, VMware_bootbank_esx-base_7.0.1-0.25.17325551, VMware_bootbank_esx-dvfilter-generic-fastpath_7.0.1-0.25.17325551, VMware_bootbank_esx-ui_1.34.4-16668064, VMware_bootbank_esx-update_7.0.1-0.25.17325551, VMware_bootbank_esx-xserver_7.0.1-0.25.17325551, VMware_bootbank_gc_7.0.1-0.25.17325551, VMware_bootbank_loadesx_7.0.1-0.25.17325551, VMware_bootbank_lsuv2-hpv2-hpsa-plugin_1.0.0-3vmw.701.0.0.16850804, VMware_bootbank_lsuv2-intelv2-nvme-vmd-plugin_1.0.0-2vmw.701.0.0.16850804, VMware_bootbank_lsuv2-lsiv2-drivers-plugin_1.0.0-4vmw.701.0.0.16850804, VMware_bootbank_lsuv2-nvme-pcie-plugin_1.0.0-1vmw.701.0.0.16850804, VMware_bootbank_lsuv2-oem-dell-plugin_1.0.0-1vmw.701.0.0.16850804, VMware_bootbank_lsuv2-oem-hp-plugin_1.0.0-1vmw.701.0.0.16850804, VMware_bootbank_lsuv2-oem-lenovo-plugin_1.0.0-1vmw.701.0.0.16850804, VMware_bootbank_lsuv2-smartpqiv2-plugin_1.0.0-4vmw.701.0.0.16850804, VMware_bootbank_native-misc-drivers_7.0.1-0.25.17325551, VMware_bootbank_vdfs_7.0.1-0.25.17325551, VMware_bootbank_vmware-esx-esxcli-nvme-plugin_1.2.0.38-1vmw.701.0.0.16850804, VMware_bootbank_vsan_7.0.1-0.25.17325551, VMware_bootbank_vsanhealth_7.0.1-0.25.17325551
   VIBs Removed: EMU_bootbank_lpfc_12.6.228.4-1OEM.700.1.0.15525992, SFC_bootbank_sfvmk_2.3.3.1016-1OEM.670.0.0.8169922, VMW_bootbank_brcmfcoe_12.0.1500.0-1vmw.700.1.0.15843807, VMW_bootbank_elxiscsi_12.0.1200.0-1vmw.700.1.0.15843807, VMW_bootbank_elxnet_12.0.1250.0-5vmw.700.1.0.15843807, VMW_bootbank_iavmd_2.0.0.1055-3vmw.700.1.0.15843807, VMW_bootbank_iser_1.1.0.0-1vmw.700.1.0.15843807, VMW_bootbank_lpnic_11.4.62.0-1vmw.700.1.0.15843807, VMW_bootbank_lsi-msgpt2_20.00.06.00-2vmw.700.1.0.15843807, VMW_bootbank_lsi-msgpt3_17.00.10.00-1vmw.700.1.0.15843807, VMW_bootbank_mtip32xx-native_3.9.8-1vmw.700.1.0.15843807, VMW_bootbank_ne1000_0.8.4-10vmw.700.1.0.15843807, VMW_bootbank_nenic_1.0.29.0-1vmw.700.1.0.15843807, VMW_bootbank_nfnic_4.0.0.44-1vmw.700.1.0.15843807, VMW_bootbank_nhpsa_2.0.50-1vmw.700.1.0.15843807, VMW_bootbank_nmlx4-core_3.19.16.7-1vmw.700.1.0.15843807, VMW_bootbank_nmlx4-en_3.19.16.7-1vmw.700.1.0.15843807, VMW_bootbank_nmlx4-rdma_3.19.16.7-1vmw.700.1.0.15843807, VMW_bootbank_ntg3_4.1.4.1-1vmw.700.1.0.15843807, VMW_bootbank_nvmxnet3-ens_2.0.0.22-1vmw.700.1.0.15843807, VMW_bootbank_nvmxnet3_2.0.0.30-1vmw.700.1.0.15843807, VMW_bootbank_pvscsi_0.1-2vmw.700.1.0.15843807, VMW_bootbank_qflge_1.1.0.11-1vmw.700.1.0.15843807, VMW_bootbank_rste_2.0.2.0088-7vmw.700.1.0.15843807, VMW_bootbank_smartpqi_1.0.4.3011-1vmw.700.1.0.15843807, VMW_bootbank_vmkata_0.1-1vmw.700.1.0.15843807, VMW_bootbank_vmkfcoe_1.0.0.2-1vmw.700.1.0.15843807, VMW_bootbank_vmkusb_0.1-1vmw.700.1.25.16324942, VMW_bootbank_vmw-ahci_1.3.9-1vmw.700.1.0.15843807, VMware_bootbank_cpu-microcode_7.0.0-1.25.16324942, VMware_bootbank_elx-esx-libelxima.so_12.0.1200.0-2vmw.700.1.0.15843807, VMware_bootbank_esx-base_7.0.0-1.25.16324942, VMware_bootbank_esx-dvfilter-generic-fastpath_7.0.0-1.25.16324942, VMware_bootbank_esx-ui_1.34.0-15603211, VMware_bootbank_esx-update_7.0.0-1.25.16324942, VMware_bootbank_esx-xserver_7.0.0-1.25.16324942, VMware_bootbank_lsuv2-hpv2-hpsa-plugin_1.0.0-2vmw.700.1.0.15843807, VMware_bootbank_lsuv2-intelv2-nvme-vmd-plugin_1.0.0-2vmw.700.1.0.15843807, VMware_bootbank_lsuv2-lsiv2-drivers-plugin_1.0.0-2vmw.700.1.0.15843807, VMware_bootbank_lsuv2-smartpqiv2-plugin_1.0.0-3vmw.700.1.0.15843807, VMware_bootbank_native-misc-drivers_7.0.0-1.25.16324942, VMware_bootbank_vmware-esx-esxcli-nvme-plugin_1.2.0.37-1vmw.700.1.0.15843807, VMware_bootbank_vsan_7.0.0-1.25.16324942, VMware_bootbank_vsanhealth_7.0.0-1.25.16324942
   VIBs Skipped: BCM_bootbank_bnxtnet_216.0.72.0-1OEM.700.1.0.15525992, BCM_bootbank_bnxtroce_216.0.65.0-1OEM.700.1.0.15525992, BCM_bootbank_dell-shared-perc8_06.806.92.00-1OEM.700.1.0.15843807, BCM_bootbank_lsi-mr3_7.713.08.00-1OEM.700.1.0.15843807, BCM_bootbank_lsi-msgpt35_15.00.00.00-1OEM.700.1.0.15843807, DEL_bootbank_dell-configuration-vib_7.0.0-A00, DEL_bootbank_dellemc-osname-idrac_7.0.0-A00, INT_bootbank_i40en_1.10.9.0-1OEM.700.1.0.15525992, INT_bootbank_igbn_1.4.11.0-1OEM.700.1.0.15525992, INT_bootbank_ixgben_1.8.9.0-1OEM.700.1.0.15525992, MEL_bootbank_nmlx5-core_4.19.70.1-1OEM.700.1.0.15525992, MEL_bootbank_nmlx5-rdma_4.19.70.1-1OEM.700.1.0.15525992, Marvell_bootbank_qlnativefc_4.1.14.0-1OEM.700.1.0.15843807, QLC_bootbank_qcnic_2.0.53.0-1OEM.700.1.0.15843807, QLC_bootbank_qedentv_3.40.19.0-1OEM.700.1.0.15843807, QLC_bootbank_qedf_2.2.8.0-1OEM.700.1.0.15843807, QLC_bootbank_qedi_2.19.9.0-1OEM.700.1.0.15843807, QLC_bootbank_qedrntv_3.40.18.0-1OEM.700.1.0.15843807, QLC_bootbank_qfle3_1.4.8.0-1OEM.700.1.0.15843807, QLC_bootbank_qfle3f_2.1.9.0-1OEM.700.1.0.15843807, QLC_bootbank_qfle3i_2.1.5.0-1OEM.700.1.0.15843807, VMware_locker_tools-light_11.1.1.16303738-16850804
[root@usd-esxi5:~]reboot
================================================================================


This should update the necessary vibs and allow vSAN to work properly after update.