Admission failure in path: init/sh is logging on fujitsu server
Article ID: 312557
Updated On:
Products
VMware vSphere ESXi
Issue/Introduction
Symptoms:
In vmkernel log, you can see below OOM logs
2019-08-18T18:05:40.123Z cpu15:816282922)MemSchedAdmit: 470: Admission failure in path: init/sh.816282924/uw.816282924
2019-08-18T18:05:40.123Z cpu15:816282922)MemSchedAdmit: 477: uw.816282924 (6502701390) extraMin/extraFromParent: 17025/17025, init (804) childEmin/eMinLimit: 53597/56320
2019-08-18T18:05:40.123Z cpu15:816282922)WARNING: LinuxThread: 423: sh: Error cloning thread: -28 (bad0081)
When you check init resource group and running prcess list, you can find 'dct_worker' process is running on init resource group
- init resource group (You can check it from log bundle)
commands$ automagicallyrun memstats -c vsi_traverse_-s.txt -r group-stats -s name:min:max:conResv:availResv:eMinPeak:rMinPeak:flags -g 816
------------------------------------------------------------------------------------------------------------------------
gid name min max conResv availResv eMinPeak rMinPeak flags
------------------------------------------------------------------------------------------------------------------------
804 init 111616 225280 11572 213708 224692 224692 ug
...
12095 sh.2100954 0 -1 1608 215868 1608 1608 uw:np:co << dct_worker process
12096 worldGroup.2100954 0 -1 0 216004 136 136 lf:pr:np:ke:sc
12097 uw.2100954 0 -1 0 216120 792 792 lf:pr:np:sc:usr
12098 vsiHeap.2100954 0 -1 0 214536 136 136 lf:pr:np:ke:sc
12099 pt.2100954 0 -1 0 214532 132 132 lf:pr:np:ke:sc
12100 cartelheap.2100954 0 -1 0 214136 276 276 lf:pr:np:ke:sc
12101 uwshmempt.2100954 0 -1 0 212932 0 0 lf:pr:np:ke:sc
12102 uwAsyncRemapHeap.2100954 0 -1 0 214132 136 136 lf:pr:np:ke:sc
commands$ less ps_-cPTgjstz.txt |grep 2100954
WID CID WorldName GID PGID SID PCID Type State Wait CPU Time Command
2100954 2100954 sh 2100954 2099760 2099760 2097905 U WAIT UPIPER 0-79 31.141138 /bin/sh /usr/lib/cim/dct_worker
- In this situation you can meet issue like below.
Stop vmkernel logging
Host operation failed
Host not responsive issue
In vmkernel log, you can see below OOM logs
2019-08-18T18:05:40.123Z cpu15:816282922)MemSchedAdmit: 470: Admission failure in path: init/sh.816282924/uw.816282924
2019-08-18T18:05:40.123Z cpu15:816282922)MemSchedAdmit: 477: uw.816282924 (6502701390) extraMin/extraFromParent: 17025/17025, init (804) childEmin/eMinLimit: 53597/56320
2019-08-18T18:05:40.123Z cpu15:816282922)WARNING: LinuxThread: 423: sh: Error cloning thread: -28 (bad0081)
When you check init resource group and running prcess list, you can find 'dct_worker' process is running on init resource group
- init resource group (You can check it from log bundle)
commands$ automagicallyrun memstats -c vsi_traverse_-s.txt -r group-stats -s name:min:max:conResv:availResv:eMinPeak:rMinPeak:flags -g 816
------------------------------------------------------------------------------------------------------------------------
gid name min max conResv availResv eMinPeak rMinPeak flags
------------------------------------------------------------------------------------------------------------------------
804 init 111616 225280 11572 213708 224692 224692 ug
...
12095 sh.2100954 0 -1 1608 215868 1608 1608 uw:np:co << dct_worker process
12096 worldGroup.2100954 0 -1 0 216004 136 136 lf:pr:np:ke:sc
12097 uw.2100954 0 -1 0 216120 792 792 lf:pr:np:sc:usr
12098 vsiHeap.2100954 0 -1 0 214536 136 136 lf:pr:np:ke:sc
12099 pt.2100954 0 -1 0 214532 132 132 lf:pr:np:ke:sc
12100 cartelheap.2100954 0 -1 0 214136 276 276 lf:pr:np:ke:sc
12101 uwshmempt.2100954 0 -1 0 212932 0 0 lf:pr:np:ke:sc
12102 uwAsyncRemapHeap.2100954 0 -1 0 214132 136 136 lf:pr:np:ke:sc
commands$ less ps_-cPTgjstz.txt |grep 2100954
WID CID WorldName GID PGID SID PCID Type State Wait CPU Time Command
2100954 2100954 sh 2100954 2099760 2099760 2097905 U WAIT UPIPER 0-79 31.141138 /bin/sh /usr/lib/cim/dct_worker
- In this situation you can meet issue like below.
Stop vmkernel logging
Host operation failed
Host not responsive issue
Environment
VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.7
Cause
This issue occurs because init process OOM(Out of Memory) and the root cause is Fujitsu's CIM provider process(dct_worker) is running on init resource group.
Init process is related with many other processes and no other 3rd party process should not be running on init resource group.
Init process is related with many other processes and no other 3rd party process should not be running on init resource group.
Resolution
It's resolved in ServerView RAID Core Provider 9.10.04.
Customer need to contact hardware vendor and resolve this issue with updating CIM provider.
Workaround:
Customer need to contact hardware vendor and get support from them.
Here is the detailed procedure to install and remove VIB.
1. Upload the CIM provider file and unzip it
2. Stop sfcbd_watchdog
/etc/init.d/sfcbd_watchdog stop
3. Install software
ex) esxcli software vib install –d /tmp/ FTS_Ser~/esxi/Fujitsu~/Fujitsu-ESX-6.~bundle-15748290.zip
4. remove previous software
esxcli software vib remove –n raid0
5. start sfcbd_watchdog
/etc/init.d/sfcbd_watchdog start
6. Check the init resource group
memstats -r group-stats mb -s'gid:name:eMinPeak:rMinPeak:consumed'|grep init
You can find the init group number from above command
memstats -r "group-stats" -s "name:min:max:eMinPeak:rMinPeak:consumed" -l 2 -g [INIT_GROUP_NUMBER]
Customer need to contact hardware vendor and resolve this issue with updating CIM provider.
Workaround:
Customer need to contact hardware vendor and get support from them.
Here is the detailed procedure to install and remove VIB.
1. Upload the CIM provider file and unzip it
2. Stop sfcbd_watchdog
/etc/init.d/sfcbd_watchdog stop
3. Install software
ex) esxcli software vib install –d /tmp/ FTS_Ser~/esxi/Fujitsu~/Fujitsu-ESX-6.~bundle-15748290.zip
4. remove previous software
esxcli software vib remove –n raid0
5. start sfcbd_watchdog
/etc/init.d/sfcbd_watchdog start
6. Check the init resource group
memstats -r group-stats mb -s'gid:name:eMinPeak:rMinPeak:consumed'|grep init
You can find the init group number from above command
memstats -r "group-stats" -s "name:min:max:eMinPeak:rMinPeak:consumed" -l 2 -g [INIT_GROUP_NUMBER]
Feedback
Yes
No
Powered by
