vCenter Virtual Machine and some of the VMs become inaccessible.

Products

• VMware vSphere ESXi

Issue/Introduction

• The issue was observed after the Storage Array Firmware upgrade.
• Connectivity is managed through the NFS protocol.
• Error message in var/run/log/vmkernel logs :

        Al (177) vmkalert: cpu49:2353923) ALERT: BC: 3042: File #########. scoreboard closed with dirty buffers. Possible data loss.
        Wa(180) vmkwarning: cpu49:2353923) WARNING: NFSLock: 4483: Unable to remove expired or lost primary lockfile .lck-la015f0100000000 for ###########. vmx.lck
        Wa(180) vmkwarning: cpu49:2353923) WARNING: NFSLock: 4483: Unable to remove expired or lost primary lockfile .lck-45035f0100000000 for ############-flat.vmdk
        Wa(180) vmkwarning: cpu2:2354208) WARNING: UserDump: 1392: vmx-vcpu-1:########: Tried to open file /vmfs/volumes/########-########/##########/vmx-zdump.000, failed with status Read only
        Wa(180) vmkwarning: cpu2:2354208) WARNING: UserDump: 1598: vmx-vcpu-1:########:##########: Failed to get core file name.

•  Error message in var/run/log/vmkwarning logs:

      Wa(180) vmkwarning: cpu28:2097379) WARNING: NFSLock: 4122: cannot remove expired lock file .lck-dd025f0100000000 : Read only
      In(182) vmkernel: cpu28:2097379) NFSLock: 5141: failed to get lock on file vmx-##################-1.vswp(.lck-dd025f0100000000) 0x430b241f8ab0 on ############### : Read o$
      Wa(180) vmkwarning: cpu28:2097379) WARNING: SwapFileOps: 601: Failed to unlink /vmfs/volumes/########-########/###########/vmx-############################-1.vswp: Read only
      In(182) vmkernel: cpu52:2100450) NFSLock: 5141: failed to get lock on file host-######-poweron(.lck-2e015f0100000000) 0x430b2423b9e0 on ###############) : Read only. fromunlink Y
      Wa(180) vmkwarning: cpu59:2100598) WARNING: NFSLock: 4012: Failed to create .lck-bb00640100000000 : Read only

 

       

Environment

VMware vSphere ESXi 8.0.x

Cause

"Dirty buffers" refers to data that was sitting in system's RAM (temporary memory) waiting to be written to the physical disk when the program or system shut down unexpectedly.

Resolution

 Contact the Storage Array Vendor for further troubleshooting on this issue.

Additional Information

Netapp Article:

https://kb.netapp.com/on-prem/ontap/DP/SnapMirror/SnapMirror-KBs/Vserver_onlined_after_Upgrade_causing_access_issue