"Failed to decrypt ciphertext: authentication tag", vMotion fails with decryption failures

Products

• VMware vSphere ESXi

Issue/Introduction

• vMotion failure with below errors in vmkernel
  
  /var/run/log/vmkernel.log

YYYY-MM-DDTHH:MM:SSZ Wa(180) vmkwarning: cpu50:3938113)WARNING: cryptoModuleLog: CryptoWrapper_AesHwGcmDecrypt:656: Failed to decrypt ciphertext: authentication tag mismatch.
YYYY-MM-DDTHH:MM:SSZ Wa(180) vmkwarning: cpu50:3938113)WARNING: VMotionEncryption:443: 2288373554472828196 D: Decryption failed: Failure.

• VM communication also might get impacted if vMotion and VM traffic is sharing the same 'vmnic'.

Environment

VMware ESXi 8.0.x

VMware ESXi 7.0.x

Cause

This is caused due to data corruption at physical layer during the data transmission.

Resolution

This issue is generally caused due to faulty hardware components such as NIC card and in-depth investigation with Packet captures will be required to identify the possible faulty components. Physical NIC can be replaced as a first step to isolate the issue. Please contact Broadcom Technical Support for further assistance.