Temporary failure in VMware environment using vSAN storage
Article ID: 409218
Updated On:
Products
Issue/Introduction
VMs running on vSAN had high latency or failures
vSAN performance graphs show a period of time with no data
vSAN health check shows multiple alarms and errors:
(This screen shot is an example and the environment may show different alarms and errors)
Environment
vSAN
Cause
One or more hosts in the vSAN cluster have lost network connection to other hosts in the cluster over the vSAN vmk port.
From the vmkernel logs you will see membership changes during the time of the event.
CMMDS: LeaderRemoveNodeFromMembership:7965: ########-####-####-####-############: Removing node ########-####-####-####-############(vsanNodeType: data) from the cluster membership
)CMMDS: CMMDSUtil_PrintArenaEntry:100: ########-####-####-####-############: [457652584]:Adding a new Membership entry (a########-####-####-####-############) with 7 members:
CMMDS: LeaderRemoveNodeFromMembership:########-####-####-####-############: Removing node ########-####-####-####-############(vsanNodeType: data) from the cluster membership
CMMDS: CMMDSUtil_PrintArenaEntry:100: ########-####-####-####-############: [457652633]:Adding a new Membership entry (########-####-####-####-############) with 6 members:Resolution
Review the cause of membership changes and network connection issues between hosts over the vSAN VMK port.
See the following for additional network troubleshooting: Troubleshooting the vSAN Network
Look for any network issues from the host side by reviewing NIC stats with the following command:
vsish –e get /net/pNics/vmnic#/stats
Example output:
vsish -e get /net/pNics/vmnic0/stats
device {
-- General Statistics:
Rx Packets:17174928
Tx Packets:0
Rx Bytes:12493488653
Tx Bytes:0
Rx Errors:0
Tx Errors:0
Rx Dropped:0
Tx Dropped:0
Rx Multicast:0
Rx Broadcast:0
Tx Multicast:0
Tx Broadcast:0
Collisions:0
Rx Length Errors:0
Rx Over Errors:0
Rx CRC Errors:0
Rx Frame Errors:0
Rx Fifo Errors:0
Rx Missed Errors:0
Tx Aborted Errors:0
Tx Carrier Errors:0
Tx Fifo Errors:0
Tx Heartbeat Errors:0
Tx Window Errors:0
Module Interface Rx packets:17174928
Module Interface Tx packets:0
Module Interface Rx dropped:0
Module Interface Tx dropped:0
-- Driver Specific Statistics:
rx_packets : 17174928
tx_packets : 0
rx_bytes : 12562055220
tx_bytes : 0
rx_broadcast : 0
tx_broadcast : 0
rx_multicast : 0
tx_multicast : 0
rx_errors : 0
tx_errors : 0
tx_dropped : 0
multicast : 0
collisions : 0
Address any issues with NIC driver / firmware or the physical hardware.
If no issues are found on the NIC cards or with the physical host's hardware, engage with the environment's physical networking team to review the physical network.
Additional Information
Feedback
