The Spherelet service is failing to start on the ESXi host after enabling execInstalledOnly.
Article ID: 320681
Updated On:
Products
Issue/Introduction
Symptoms:
- The execInstalledOnly is a security option used to secure the ESXi hosts against ransomware execution. After you enable this option in vSphere 7, the Spherelet service fails to start.
[root@esxi:~] /etc/init.d/spherelet status
YYYY-MM-DD 10:41:22,688 init.d/spherelet spherelet init script invoked via the following hierarchy
YYYY-MM-DD 10:41:22,688 init.d/spherelet XXX6086: -sh
YYYY-MM-DD 10:41:22,688 init.d/spherelet XXX5949: sshd -i
YYYY-MM-DD 10:41:22,688 init.d/spherelet 2098918: /usr/lib/vmware/busybox/bin/busybox inetd /var/run/inetd.conf
YYYY-MM-DD 10:41:22,688 init.d/spherelet 2097930: /bin/init
YYYY-MM-DD 10:41:22,840 init.d/spherelet Log fetcher support: True
YYYY-MM-DD 10:41:22,856 init.d/spherelet spherelet is not running
YYYY-MM-DD 10:41:22,856 init.d/spherelet spherelet is not running
- When you check the vmkernel.log, you find:
YYYY-MM-DDTHH:MM:SS.663Z cpu51:XXX7674)SchedVsi: 2083: Group: host/vim/vmvisor/spherelet(750): min=112 max=4509, units: mb
YYYY-MM-DDTHH:MM:SS.663Z cpu51:XXX7674)SchedVsi: 2083: Group: host/vim/vmvisor/spherelet/imgfetcher(751): min=2700 max=2700, units: mb
YYYY-MM-DDTHH:MM:SS.663Z cpu51:XXX7674)SchedVsi: 2083: Group: host/vim/vmvisor/spherelet/logfetcher(752): min=470 max=470, units: mb
YYYY-MM-DDTHH:MM:SS.748Z cpu78:XXX7695)UserMem: 7594: mmap denied: boot opt execInstalledOnly is set: file not installed
YYYY-MM-DDTHH:MM:SS.831Z cpu2:XXX7710)UserMem: 7594: mmap denied: boot opt execInstalledOnly is set: file not installed
YYYY-MM-DDTHH:MM:SS.906Z cpu19:XXX7723)UserMem: 7594: mmap denied: boot opt execInstalledOnly is set: file not installed
- If the host was in maintenance mode, exiting the maintenance mode on the vCenter is stuck at 95%.
Environment
VMware vSphere 7.0 with Tanzu
Cause
- When the execInstalledOnly boot option is enabled, the Go runtime fails to get initialised because of mmap() denial. Hence, the Golang binaries like spherelet are failing to start.
Resolution
- vSphere 8 isn't impacted by this issue. The VMware engineering team initially plans to cross-port the fix in ESXi 7 P08 (ESXi-7.0U3o-22348816).
Workaround:
- If the PodVMs or embedded Harbor aren't in use, just restart the vpxd service on the vCenter to kill the exit maintenance mode task stuck at 95%.
service-control --stop vmware-vpxd && service-control --start vmware-vpxd
- If the PodVMs are in use, temporarily disable the execInstalledOnly until the fix is released.
Commands to disable execInstalledOnly
To disable the kernel option$ esxcli system settings kernel set -s execInstalledOnly -v FALSE
validate$ esxcli system settings kernel set -o execInstalledOnly
Note: ESXi has different execInstalledOnly options, hence please validate if both the options are disabled:
Runtime option (Introduced for ESXi 7.0, as a soft enablement)# localcli system settings advanced list -o /User/ExecInstalledOnly
Legacy / vmkernel boot option - provides a stronger protection than the runtime option# localcli system settings kernel list -o execInstalledOnly
Additional documents to enable or disable execInstalledOnly
Additional Information
Feedback
