Vxrail Hosts going into not responding state due to perccli vibs
search cancel

Vxrail Hosts going into not responding state due to perccli vibs

book

Article ID: 326474

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

Symptoms:
Hosts going into not responding state randomly in a Vxrail environment. SSH , esxcli commands would not run.  Hosts running on 6.7 update 1 Build 11675023

Any commands run on affected hosts was giving errors -:

"cannot open shared object file Error 24ps: error while loading shared libraries: libz.so.1: cannot open shared object file : Error 24"

OSError : [Errno 24] Too many open files: '/lib64/python3.5/site-packages/vmsupport/encryption.pyc
  • The "esxcli "commands returns following errors -:
Got no data from process: python /usr/lib/vmware/vsan/esxcli/esxcli_vsan_debug.pyc sync.summary
  • The vmkernel logs filled with -:
Open file limit 65536 exceeded
  • A large number of "percli" processes running as per uwstats and ps commands.

#uwstats | grep -i perccli
58802654         perccli.64201508    4      w   7   8  256   0   0       2   2  4096
58802683         perccli.64201550    4      w   7   8  256   0   0       2   2  4096
58802772         perccli.64201664    4      w   7   8  256   0   0       2   2  4096
58802806         perccli.64201716    4      w   7   8  256   0   0       2   2  4096
58802896         perccli.64201827    4      w   7   8  256   0   0       2   2  4096
58802966         perccli.64201912    4      w   7   8  256   0   0       2   2  4096
58803099         perccli.64202075    4      w   7   8  256   0   0       2   2  4096
58803469         perccli.64202518    4      w   7   8  256   0   0       2   2  4096
58803557         perccli.64202629    4      w   7   8  256   0   0       2   2  4096
 
  • vsanmgmt.log was filled with following errors -:
2019-08-14T02:42:56Z VSANMGMTSVC: ERROR vsanperfsvc[38a43808-be3d-11e9] [VsanEsxHclUtil::__init__] Failed to run tool perccli: Running storcli failed: b'/opt/lsi/perccli/perccli: error while loading shared libraries: libm.so.6: cannot open shared object file: Error 24\n'
2019-08-14T02:44:01Z VSANMGMTSVC: INFO vsanperfsvc[5f5097e2-be3d-11e9] [VsanStorcli::_runStorcli] Running: /opt/lsi/perccli/wrapperperccli ++group=host/vim/tmp /call show J
2019-08-14T02:44:01Z VSANMGMTSVC: ERROR vsanperfsvc[5f5097e2-be3d-11e9] [VsanEsxHclUtil::__init__] Failed to run tool perccli: Running storcli failed: b'/opt/lsi/perccli/perccli: error while loading shared libraries: libm.so.6: cannot open shared object file: Error 24\n'
2019-08-14T02:46:12Z VSANMGMTSVC: INFO vsanperfsvc[ad4a8080-be3d-11e9] [VsanStorcli::_runStorcli] Running: /opt/lsi/perccli/wrapperperccli ++group=host/vim/tmp /call show J

2019-08-14T02:46:32Z VSANMGMTSVC: ERROR vsanperfsvc[ad4a8080-be3d-11e9] [VsanEsxHclUtil::__init__] Failed to run tool perccli: Exception 'RunCommandError' occured running command '['/opt/lsi/perccli/wrapperperccli', '++group=host/vim/tmp', '/call', 'show', 'J']'
2019-08-14T02:47:17Z VSANMGMTSVC: INFO vsanperfsvc[d42a7ffe-be3d-11e9] [VsanStorcli::_runStorcli] Running: /opt/lsi/perccli/wrapperperccli ++group=host/vim/tmp /call show J
2019-08-14T02:52:43Z VSANMGMTSVC: INFO vsanperfsvc[9685f45a-be3e-11e9] [VsanStorcli::_runStorcli] Running: /opt/lsi/perccli/wrapperperccli ++group=host/vim/tmp /call show J
2019-08-14T02:52:43Z VSANMGMTSVC: ERROR vsanperfsvc[9685f45a-be3e-11e9] [VsanEsxHclUtil::__init__] Failed to run tool perccli: Running storcli failed: b'/opt/lsi/perccli/perccli: error while loading shared libraries: libgcc_s.so.1: cannot open shared object file: Error 24\n'
2019-08-14T02:54:54Z VSANMGMTSVC: INFO vsanperfsvc[e41310fe-be3e-11e9] [VsanStorcli::_runStorcli] Running: /opt/lsi/perccli/wrapperperccli ++group=host/vim/tmp /call show J
2019-08-14T02:54:54Z VSANMGMTSVC: ERROR vsanperfsvc[e41310fe-be3e-11e9] [VsanEsxHclUtil::__init__] Failed to run tool perccli: Running storcli failed: b'/opt/lsi/perccli/perccli: error while loading shared libraries: libgcc_s.so.1: cannot open shared object file: Error 24\n'

Cause

Incompatible perccli vibs running on ESXi hosts in the Vxrail vsan cluster. 
The hosts were running on 330mini adapter which does not need the perccli vib.
Incompatible Vxrail version .

Resolution

  • The issue is resolved in the VxRail version 4.7.211 and later ( confirmed by DELL-EMC ) 


Workaround:
Uninstall perccli vibs from all the ESXi hosts in the Vxrail vsan cluster.

# esxcli software vib remove -n vmware-perccli-007.0529.0000.0000
Removal Result
 Message: Operation finished successfully.
 Reboot Required: false
  VIBs Installed:
  VIBs Removed: LSI_bootbank_vmware-perccli-007.0529.0000.0000_007.0529.0000.0000-01
  VIBs Skipped:

Additional Information





Impact/Risks:
Hosts randomly going into not responding state may impact production (CPU, memory, and storage) and cause DU (Data Unavailability) situation.

Powered by Wolken Software