[VMConAWS] VMware Cloud Flex Storage intermittently disconnects from ESXi host

Products

• VMware Cloud on AWS

Issue/Introduction

Indentify events leading to intermittent connectivity from the host to the Cloud Flex storage and confirming the known issue

Symptoms:
Cloud flex storage disconnects intermittently from ESXi hosts. 

Latency events are seen in the hosts 

/var/log/vmkwarning.log or /var/log/vobd.log

vmkwarning: cpu55:2098561)WARNING: NFS: 5211: NFS volume flexstorage-1 performance has deteriorated. I/O latency increased from average value of 2826(us) to 1785611(us).
vmkwarning: cpu55:2098561)WARNING: NFS: 5211: NFS volume flexstorage-1 performance has deteriorated. I/O latency increased from average value of 2832(us) to 3747725(us).
vmkwarning: cpu55:2098560)WARNING: NFS: 5211: NFS volume flexstorage-1 performance has deteriorated. I/O latency increased from average value of 2832(us) to 7827077(us).
 vmkwarning: cpu33:2098561)WARNING: NFS: 5211: NFS volume flexstorage-1 performance has deteriorated. I/O latency increased from average value of 2833(us) to 15695947(us). 

Following APD events are seen following latency events 

/var/log/vobd.log

vobd[2098017]:  [APDCorrelator] 93243295065us: [vob.storage.apd.exit] Device or filesystem with identifier [e0417df9-70e8d904] has exited the All Paths Down state.2023-08-11
vobd[2098017]:  [APDCorrelator] 93235963575us: [esx.problem.storage.apd.start] Device or filesystem with identifier [e0417df9-70e8d904] has entered the All Paths Down state.2023-08-11
vobd[2098017]:  [APDCorrelator] 93236993019us: [vob.storage.apd.start] Device or filesystem with identifier [e0417df9-70e8d904] has entered the All Paths Down state.2023-08-11

/var/log/hostd.log

Hostd[2103122]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 7040 : Device or filesystem with identifier e0417df9-70e8d904 has entered the All Paths Down state.2023-08-11

Cause

This is known issue affecting Flex storage versions v25 and older.

Resolution

This is a known issue and is fixed in Cloud Flex storage version v26. 

You may configure "Host Storage Status" alarm for the cluster in vCenter to monitor the events. Refer vSphere documentation to set an alarm.

Additional Information

Impact/Risks:
Lose of connectivity to NFS mounts