ESXi 4.1 virtual machine fails to power on with the error: FoundryVMDirectlyOpenSocketToVMX: Failed to create socket pair
search cancel

ESXi 4.1 virtual machine fails to power on with the error: FoundryVMDirectlyOpenSocketToVMX: Failed to create socket pair

book

Article ID: 301999

calendar_today

Updated On:

Products

VMware

Issue/Introduction

Symptoms:

When using VMware ESXi 4.1, you may experience one or more of these symptoms:

  • Unable to connect to the MKS
  • Unable to power on virtual machines
  • Powering on virtual machines reports some virtual machines in an invalid state
  • Power state of the virtual machine is reported incorrectly in the vCenter Server/ESX inventory
  • If you restart hostd, virtual machines that were previously not in an invalid state may appear as invalid
  • When a virtual machine is in an invalid state, you see these errors:

    vmx| VmdbPipeStreamsOvlError: write failed, draining reads
    vmx| VmdbPipeStreamsOvlError: Couldn't initiate write
    vmx| Redirecting stdin/stdout/stderr to /dev/null.
    vmx| SOCKET 1 (91) send error 12: Cannot allocate memory
    vmx| Vix: [177936 mainDispatch.c:2472]: VMAutomation: Connection Error (1) on connection 0.
    vmx| SOCKET 1 (91) send error 12: Cannot allocate memory -> not able to create socket,no mem available.
    mks| SOCKET 2 (92) recv error 104: Connection reset by peer
    mks| SOCKET 2 (92) destroying VNC backend on socket error: 1

  • The messages log contains entries similar to:

    • sfcb-CIMXML-Processor[9857708]: SendMsg sending to 7 9857708-9 Bad file descriptor
      sfcb-CIMXML-Processor[9857708]: spSendMsg sending to 7 9857708-9 Bad file descriptor
      sfcb-CIMXML-Processor[9857708]: --- spSendReq/spSendMsg failed to send on 7 (-1)
      root: sfcbd-watchdog:Restarting SFCB! Log a bug!!!
      root: sfcbd-watchdog:stopping sfcbd
      root: sfcbd Stopping sfcbd
      root: sfcbd-watchdog:starting sfcbd
      root: sfcbd Starting sfcbd
      sfcb-sfcb[9849840]: --- Using /etc/sfcb/sfcb.cfg

    • FoundryVMDirectlyOpenSocketToVMX: Failed to create socket pair.

  • The hostd logs may report this error when you try to power on the virtual machine:

    Cannot connect to /var/run/vmware/root_0/1299674323658606_59831734/testAutomation-fd: File not found

  • vMotion fails intermittently at a random percentage
  • Connecting to the remote console via the vSphere Client fails with the error:

    Unable to connect to the MKS: There is no VMware process running for config file

  • Unable to retrieve any files from the datastore via Datastore Browser
  • Copying a file using the datastore browser gives the error:

    Failed to log into NFC server

  • The vm-support diagnostic information gathering utility is unresponsive
  • The Health Status tab may not load correctly
  • Powering on a virtual machine fails at 95% with the error:

    A general system error occurred: Cannot connect to the virtual machine


Cause

This issue occurs due to exhaustion of VMkernel socket resources on ESXi hosts. It occurs more frequently on hosts where OEM CIM providers have been installed, which may be caused by additional CIM Providers loaded under sfcbd. To ensure the stability of the OEM CIM providers, ensure that they are updated to the latest release.
Note: This issue only occurs on ESXi. It does not occur on ESX. On ESX, sfcbd runs as a process in the service console rather than from a world under the VMkernel.

Resolution

Resolution

This issue is resolved by VMware ESXi 4.1 Patch ESXi410-201107401-BG. For more information, see VMware ESXi 4.1 Patch ESXi410-201107401-BG: Updates Firmware (2000609).
Note: A system which is found to be in a bad state must be rebooted prior to applying the patch.

Workaround

To work around this issue, stop the sfcbd hardware monitoring agent on the ESXi host. When sfcbd is disabled, hardware status information for the ESXi host will be unavailable.
To stop sfcbd:
  1. Log into the VMware ESXi host as the root user. For more information, see Using Tech Support Mode in ESXi 4.1 and ESXi 5.0 (1017910).

  2. Run the command:

    /etc/init.d/sfcbd-watchdog stop

  3. To make this change persistent on reboot, run these commands:

    chkconfig sfcbd-watchdog off
    chkconfig sfcbd off
If hardware monitoring is an environmental requirement, you can extend the amount of time before the issue re-occurs by changing the configuration of sfcbd:
  1. From the ESXi shell, edit the /etc/sfcb/sfcb.cfg file using a text editor.
  2. Search for the entry provProcs: 16, and change the value from 16 to 12.
  3. Restart sfcbd for the changes to take effect using the command:

    /etc/init.d/sfcbd-watchdog restart

    Note: Depending on the system workload, this change may only temporarily resolve the issue.

If the symptoms persist after applying this patch or workaround, collect logs from the environment and contact VMware Support. For more information, see:



Additional Information

Virtual machines can appear as powered off while they are actually powered on. You can recover these virtual machines without rebooting them. For more information, see Virtual machine appears powered off after restarting the management services on the host (1033591).
To be alerted when this document is updated, click the Subscribe to Article link in the Actions box
“vm-support” command in ESX/ESXi to collect diagnostic information
Using Tech Support Mode in ESXi 4.1, ESXi 5.x, and ESXi 6.x
VMware ESXi 4.1 Patch Release ESXi 410-201011001
Virtual machine appears powered off after restarting the management services on the host
VMware ESXi 4.1, Patch Release ESXi410-201104001
VMware ESX 4.1 Patch ESX410-201107401-BG: Updates VMkernel, VMX, CIM
VMware ESXi 4.1 Patch ESXi410-201107401-BG: Updates Firmware
How to file a Support Request in Customer Connect
ESXi 4.1 仮想マシンが次のエラーでパワーオンに失敗する: FoundryVMDirectlyOpenSocketToVMX: Failed to create socket pair