VMotion fails after a third-party security tool performs a port scan of the ESX/ESXi hosts

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• VMotion fails after a third-party security tool (such as IBM Internet Security Systems) performs a port scan of the ESX or ESXi hosts.
• You see errors similar to:
  
  cpu1:1086) Migrate: 2250 Error with migration listen socket, shutting down: I/O error.
  A general system error occurred; timed out waiting for migration data
  
  
• The vmkernel.log contains messages similar to:
  
  May 1 21:20:45 ESXsrvr vmkernel: 11:22:29:39.358 cpu13:1915)World: vm 1923: 901: Starting world migSendHelper-1916 with flags 1
  May 1 21:20:45 ESXsrvr vmkernel: 11:22:29:39.358 cpu13:1915)World: vm 1924: 901: Starting world migRecvHelper-1916 with flags 1
  May 1 21:20:45 ESXsrvr vmkernel: 11:22:29:39.364 cpu1:1086)MigrateNet: vm 1086: 854: Accepted connection from <xxx.xxx.xxx.xxx>
  May 1 21:21:05 ESXsrvr vmkernel: 11:22:29:59.642 cpu12:1916)Migrate: 7309: 1241227232551280: Another pre-copy iteration needed with 30737 modified pages (last = -1)
  May 1 21:21:07 ESXsrvr vmkernel: 11:22:30:02.092 cpu10:1916)Migrate: 7309: 1241227232551280: Another pre-copy iteration needed with 17783 modified pages (last = 30737)
  May 1 21:21:09 ESXsrvr vmkernel: 11:22:30:03.938 cpu9:1916)Migrate: 7304: 1241227232551280: Stopping pre-copy: Not enough forward progress (Modified pages 17783 vs. 22217) - stopping pre-copy
  May 1 23:32:52 ESXsrvr vmkernel: 12:00:41:45.964 cpu1:1086)MigrateNet: vm 1086: 854: Accepted connection from <xxx.xxx.xxx.xxx>
  May 1 23:32:52 ESXsrvr vmkernel: 12:00:41:45.964 cpu1:1086)WARNING: MigrateNet: vm 1086: 865: Couldn't set nodelay option on socket
  May 1 23:32:52 ESXsrvr vmkernel: 12:00:41:45.964 cpu1:1086)ALERT: Migrate: 2250: Error with migration listen socket, shutting down: I/O error
  May 1 23:32:52 ESXsrvr vmkernel: 12:00:41:45.964 cpu1:1086)Migrate: 2312: Exit requested...
  
  
• The Hostd.log contains messages similar:
  
  [2009-05-01 23:31:41.190 'App' 22911920 error] SSLStreamImpl::BIORead ( A6A2D10) failed: Connection reset by peer
  [2009-05-01 23:31:41.190 'App' 22911920 error] SSLStreamImpl::DoServerHandshake ( A6A2D10) SSL_accept failed with BIO Error
  [2009-05-01 23:31:41.190 'Proxysvc' 22911920 warning] SSL Handshake on client connection failed for peer , error=SSL Exception: BIO Error
  [2009-05-01 23:32:22.994 'App' 21588912 error] SSLStreamImpl::DoServerHandshake ( A6B9AA8) SSL_accept failed with Unexpected EOF
  [2009-05-01 23:32:22.994 'Proxysvc' 21588912 warning] SSL Handshake on client connection failed for peer <xxx.xxx.xxx.xxx>, error=SSL Exception: Unexpected EOF
  [2009-05-01 23:32:52.085 'ha-eventmgr' 130374576 info] Event 271 : Issue detected on ESXsrvr.mydomain.com in ha-datacenter: Migrate: 2250: Error with migration listen socket, shutting down: I/O error (12:00:41:45.964 cpu1:1086)

Environment

VMware ESX Server 3.5.x
VMware ESXi 4.0.x Embedded
VMware ESX 4.0.x
VMware ESXi 4.0.x Installable

Resolution

This issue is resolved in ESX/ESXi 4.0 Update 2. For more information, see vSphere 4 download page.
This issue is resolved in ESX and ESXi 3.5. For more details, see KB 1026126 (ESX) at http://kb.vmware.com/kb/1026126, and KB 1026138 (ESXi) at http://kb.vmware.com/kb/1026138.

This issue might occur if a network port-scanner-process attempts to engage VMotion migration port (8000) on the ESX or ESXi host. On ESX/ESXi 3.5.x, you must disable and then re-enable VMotion on the ESX/ESXi host.

The workaround provided to resolve the issue was:

To disable and re-enable VMotion:

1. Select the ESX/ESXi host in the VI Client.
2. Select Configuration > Advanced Settings > Migrate > Migrate.enabled.
3. Change the value of Migrate.enabled setting from 1 to 0.
4. Click OK.
5. Select Configuration > Advanced Settings > Migrate > Migrate.enabled.
6. Change the Migrate.enabled setting from 0 to 1.
7. Click OK.

To prevent VMotion from failing, you must exclude port 8000 in your port scanning software.

Note: A VMotion network should never be accessible by untrusted sources. You must isolate the management network as described in the VMware Infrastructure 3 Security Hardening Guide.