Host connection and power state during backup

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• vCenter reported "Host connection and power state" Alarm on host
• Backup is running on the host and may report error on backup software
• Hostd crashs and dumps are created before the alarm is reported
• Huge number of Blklistsvc log over the backing up VM can be found in hostd
  

  Hostd[45076243]: [Originator@6876 sub=Blklistsvc opID=xxxx sid=xxxx user=vpxuser:<BACKUP-USER>] Block list request received for snapshot <snapshot-name>, disk [] <a vmdk>, r/w: no, session xxxx
  Hostd[45076243]: [Originator@6876 sub=DiskLib opID=xxxx sid=xxxx user=vpxuser:<BACKUP-USER>] DISKLIB-VMFS  : "<a vmdk>" : failed to open (Reached the host's limit for open files): Directory open failed on '<VM PATH>': Too many open files (24). Type 3
  Hostd[45076243]: [Originator@6876 sub=DiskLib opID=xxxx sid=xxxx user=vpxuser:<BACKUP-USER>] DISKLIB-LINK  : DiskLinkOpen: Failed to open '<a vmdk>': : Reached the host's limit for open files
  Hostd[45076243]: [Originator@6876 sub=DiskLib opID=xxxx sid=xxxx user=vpxuser:<BACKUP-USER>] DISKLIB-CHAIN : DiskChainOpen: "<a vmdk>": failed to open: Reached the host's limit for open files.
  Hostd[45076243]: [Originator@6876 sub=DiskLib opID=xxxx sid=xxxx user=vpxuser:<BACKUP-USER>] DISKLIB-LIB   : Failed to open '<a vmdk>' with flags 0xe Reached the host's limit for open files (40).
  Hostd[45076243]: [Originator@6876 sub=Blklistsvc opID=xxxx sid=xxxx user=vpxuser:<BACKUP-USER>] Failed to open virtual disk: [] /<a vmdk> Reached the host's limit for open files 40.
  Hostd[45076243]: [Originator@6876 sub=Blklistsvc opID=xxxx sid=xxxx user=vpxuser:<BACKUP-USER>] Destructor for Lease <snapshot-name>-lease-3181 called

Environment

ESXi 

Cause

Too Many directly API call DiskManager.acquireLeaseExt from third-party software to the same disk.

Resolution

Restart hostd and vpxa service on the issue host.