random vmon services status changed from green to unknown
Article ID: 393992
Updated On:
Products
Issue/Introduction
・vCenter Server issue alarms that random vmon service status such as wcp, vsan-health, vlsm changed from green to unknown and back to green.
wcp status changed from green to unknown
・vmware-analytics service is Stopped.
Running:
applmgmt lookupsvc lwsmd observability observability-vapi pschealth vlcm vmafdd vmcad vmdird vmonapi vmware-certificateauthority vmware-certificatemanagement vmware-cis-license vmware-content-library vmware-eam vmware-envoy vmware-hvc vmware-infraprofile vmware-perfcharts vmware-pod vmware-postgres-archiver vmware-rhttpproxy vmware-sca vmware-sps vmware-statsmonitor vmware-stsd vmware-topologysvc vmware-trustmanagement vmware-updatemgr vmware-vapi-endpoint vmware-vdtc vmware-vmon vmware-vpostgres vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-ui vstats vtsdb wcp
Stopped:
vmcam vmware-analytics vmware-imagebuilder vmware-netdumper vmware-rbd-watchdog vmware-vcha
・From vmon.log excerpt, health checks were failed because the user was not found or Too many open files.
Er(02) host-XXXX <wcp-healthcmd> stderr pipe creation failed. Too many open files
Er(02) host-XXXX [Child XXXX] Entry corresponding to user vsan-health not found
In(05) host-XXXX <event-pub> Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonEventPublisher.py --eventdata vsan-health,UNKNOWN,HEALTHY,1
Er(02) host-XXXX [Child XXXX] Entry corresponding to user updatemgr not found
In(05) host-XXXX <event-pub> Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonEventPublisher.py --eventdata updatemgr,UNKNOWN,HEALTHY,1
Er(02) host-XXXX [Child XXXX] Entry corresponding to user wcp not found
In(05) host-XXXX <event-pub> Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonEventPublisher.py --eventdata wcp,UNKNOWN,HEALTHY,1
Er(02) host-XXXX [Child XXXX] Entry corresponding to user wcp not found
In(05) host-XXXX <event-pub> Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonEventPublisher.py --eventdata wcp,UNKNOWN,HEALTHY,1
Er(02) host-XXXX [Child XXXX] Entry corresponding to user wcp not found
In(05) host-XXXX <event-pub> Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonEventPublisher.py --eventdata wcp,UNKNOWN,HEALTHY,1
Er(02) host-XXXX [Child XXXX] Entry corresponding to user wcp not found
In(05) host-XXXX <event-pub> Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonEventPublisher.py --eventdata wcp,UNKNOWN,HEALTHY,1
Er(02) host-XXXX <vlcm-healthcmd> stderr pipe creation failed. Too many open files
In(05) host-XXXX <event-pub> Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonEventPublisher.py --eventdata vlcm,UNKNOWN,HEALTHY,1
Er(02) host-XXXX [Child XXXX] Entry corresponding to user wcp not found
In(05) host-XXXX <event-pub> Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonEventPublisher.py --eventdata wcp,UNKNOWN,HEALTHY,1
・File descriptor count of vmon is almost 1024.
ls -l /proc/[0-9]*/fd
--
/proc/XXXX 1002
--
Cause
If vmware-analytics service was stopped, vmon service's file descriptor will leak and vmon can't get service status.
Resolution
File descriptor leak will be fixed in future release.
Workaround is reboot the vCenter Server to release file descriptors and run vmware-analytics for precaution.
Feedback
