Virtual machine residing on vVol datastores go unresponsive during log collection.

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Communication to vVol datastore is lost during ESXi log collection task.
• Virtual machine and hosts go unresponsive during log collection.

Environment

VMware vSphere ESXi 7.0

Cause

- An optimization (SCSIBlockUnsupportedOpcodesAndPages) to VMware’s Pluggable Storage Architecture (PSA) was introduced in ESXi 7.0 U3 which is designed to prevent issuing commands to Storage Devices after receiving a SCSI Command failure with an Invalid Operation Code.

- For the impacted vVol Datastores, we

can see that Protocol Endpoint (PE) Devices previously failed SCSI Commands

with error code “0x5 0x20” which translates to ILLEGAL REQUEST / INVALID

COMMAND OPERATION CODE –

 

- Due to the newer SCSIBlockUnsupportedOpcodesAndPages optimization,

these devices are flagged with an Unsupported status which

is then cached persistently which means that all future SCSI command attempts

will fail, thus resulting in the vVol Datastore Read/Write failures we

observed. 

Resolution

Issue is fixed in ESXi version 7.0 Update 3f 20036589

Workaround:

- This optimization can be disabled by following the steps below –

1. Run the following

command to disable

the SCSIBlockUnsupportedOpcodesAndPages feature

-

 

# esxcfg-advcfg -s 0

/Scsi/SCSIBlockUnsupportedOpcodesAndPages

2. Get a list of all vVol Protocol

Endpoint Devices by running

the following command -

 

# esxclistorage core device

list | grep 'Is VVOL PE\| Display Name'|grep true -B1

3. For each of the devices identified in

Step # 2, run the following

VSISH command to clear the cached Unsupported

status, replacing the

"Device Name" to reflect each individual

Protocol Endpoint -

 

# vsish -e set

/storage/scsifw/devices/<DEVICE NAME>/resetOpcodeAndPageBitmaps 1

Additional Information

Impact/Risks:

Virtual machines running on vVol datastore gets hung during ESXi log collection task.