Error: "A general system error occurred: Sandboxd call timed out" when performing a vMotion task

Products

• VMware vSphere ESXi 8.0

Issue/Introduction

• After an upgrade of ESXi hosts to 8.0.x, vMotions fail with an error "A general system error occurred: Sandboxd call timed out".
  
  
• ESXi - /var/log/hostd.log
  

  yyyy-mm-ddThh:mm:ss.us Wa(164) Hostd[2745240]: [Originator@6876 sub=IO.Connection] Failed to connect; <io_obj p:0x0000002c3a809660, h:60, <UNIX ''>, <UNIX 'vmwLocalSocketSandboxVapi'>>, e: 2(No such file or directory), duration: 0msec
  yyyy-mm-ddThh:mm:ss.us Wa(164) Hostd[2745240]: [Originator@6876 sub=HttpConnectionPool-000000] Failed to get pooled connection; <cs p:0000002c111aff00, PIPE:vmwLocalSocketSandboxVapi>, (null), duration: 0msec, N7Vmacore15SystemExceptionE(No such file or directory)
  

• ESXi - /var/log/sandboxd.log
  

  yyyy-mm-ddThh:mm:ss.us In(166) sandboxd[2098186]: --> [context]zKq7AVICAgAAAP////8Jc2FuZGJveGQAAOPJR2xpYnZtYWNvcmUuc28AAL4fPQBLLywAlDIsAB7bLADg/ywAO1BSAVJ4AGxpYnB0aHJlYWQuc28uMAACP1IPbGliYy5zby42AA==[/context]
  yyyy-mm-ddThh:mm:ss.us In(166) sandboxd[2098227]: [Originator@6876 sub=Default.HTTPService.HttpConnection] HTTP Connection has timed out while waiting for further requests; <io_obj p:0x000000c854a4d0f8, h:-1, <UNIX 'vmwLocalSocketSandboxVapi'>, <UNIX 'vmwLocalSocketSandboxVapi'>>, N7Vmacore16TimeoutExceptionE(Operation timed out: Stream: <io_obj p:0x000000c854a4d0f8, h:-1, <UNIX 'vmwLocalSocketSandboxVapi'>, <UNIX 'vmwLocalSocketSandboxVapi'>>, duration: 00:00:46.386261 (hh:mm:ss.us))

• Reviewing the chkconfig.db file displays the sandboxd service is off
  
  cat chkconfig.db | grep sandboxd
/etc/init.d/sandboxd    off
  
  
• Reviewing the inittab file displays "backup.sh 1"
  
  grep backup /etc/inittab
::shutdown:/bin/backup.sh 1

Cause

The direct cause of the vMotion failure is that the sandboxd service is not running. This happens because ConfigStore enters a state where software changes fail to be recorded, preventing the necessary ESX upgrade modules from running.

This issue stems from how the host handles the backup.sh script following a Live-install. The backup.sh script runs in two modes:

• backup.sh 0: Persists any configuration changes done to the ESXi host to the disk.

• backup.sh 1: Persists configuration changes, but performs additional tasks because 1 implies the host is being shut down (this is why inittab uses 1 under the shutdown context).

The code for backup.sh is written so that the Live-install status is updated in ConfigStore when the script is run with shutdown=1. However, if backup.sh 1 is run manually and the ESXi host does not go through a boot cycle immediately afterward, ConfigStore is prepped for a reboot that hasn't happened yet.

If an upgrade is later triggered by an SDDC-M workflow, ConfigStore is stuck in a state where backup.sh 1 and all future software changes are ignored. Because backup.sh was manually run in shutdown mode after a live-install without an actual reboot cycle, the subsequent ESX upgrade modules do not run, leaving sandboxd disabled.

Resolution

This is a known issue, and currently, there is no official resolution.

Workaround

To work around this issue and allow vMotions to succeed, manually start and enable the sandboxd service.

1. To start the sandboxd service, run the following command:
   
   /etc/init.d/sandboxd start
   
   
2. To persist the sandboxd change so that it persists future reboots, run:
   
   /bin/chkconfig sandboxd on