Cold and hot Storage vMotion fails between two sites: Cannot connect to host

Products

• VMware vSphere ESXi

Issue/Introduction

• This KB addresses a particular behaviour caused by WAN traffic optimizers, that is only discovered by inspecting the TCP options of particular packets in a packet trace capturing the issue.
• The VM migration attempt fails instantly: Cannot connect to host
• Although ping is successful and MTU is consistent, the following authd banner is not displayed when probing the destination IP with nc -v IP 902 from ESXCLI:
  220 VMware Authentication Daemon Version 1.10: SSL Required, ServerDaemonProtocol:SOAP, MKSDisplayProtocol:VNC , VMXARGS supported, NFCSSL supported/t
  
  
• When capturing the traffic of a netcat against 902, the TCP Timestamp Option (8) is missing.
  Sample: Please note on the destination the timestamp "TS val 1824034367" is being set in packet 2, but on the source it is missing in packet 2.
  
  $ tcpdump -r dstvmk0_dir2.pcapng port 902


  15:38:45.916527 IP #.#.#.16.35083 > #.#.#.124.ideafarm-door: Flags [S], seq 838613922, win 65535, options [mss 1420,nop,wscale 9,sackOK,TS val 3874589256 ecr 0,unknown-33 0x21286f7f21b6af010000], length 0
  15:38:45.916558 IP #.#.#.124.ideafarm-door > #.#.#.16.35083: Flags [S.], seq 2676812151, ack 838613923, win 65535, options [mss 1420,nop,wscale 9,sackOK,TS val 1824034367 ecr 3874589256], length 0
  15:38:45.928560 IP #.#.#.16.35083 > #.#.#.124.ideafarm-door: Flags [.], ack 1, win 127, length 0
  15:38:45.928584 IP #.#.#.124.ideafarm-door > #.#.#.16.35083: Flags [R], seq 2676812152, win 0, length 0

    $ tcpdump -r srcvmk0_dir2.pcapng port 902

    15:38:45.911315 IP #.#.#.16.35083 > #.#.#.124.ideafarm-door: Flags [S], seq 838613922, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val 3874589256 ecr 0], length 0
    15:38:45.924275 IP #.#.#.124.ideafarm-door > #.#.#.16.35083: Flags [S.], seq 3750553975, ack 838613923, win 5840, options [mss 1420,nop,nop,sackOK,nop,wscale 8], length 0
    15:38:45.924337 IP #.#.#.16.35083 > #.#.#.124.ideafarm-door: Flags [.], ack 1, win 130, length 0
    15:38:45.936332 IP #.#.#.124.ideafarm-door > #.#.#.16.35083: Flags [R.], seq 1, ack 1, win 23, length 0

    Note: The preceding excerpts are only examples. Date,time and environmental variables may vary depending on your environment.

Environment

VMware vSphere ESXi 6.7
VMware vSphere ESXi 7.0

Cause

If the timestamp option gets attached (SYN) from source to destination, but filtered out on the way from destination to source (SYN-ACK), the ACK arriving at the destination acknowledging the SYN-ACK will not have that timestamp option set as per protocol standard. That leads to ESXi responding with a RST (Reset), effectively dropping the connection.

The filtering of the timestamp option can occur due to settings in a firewall, router or WAN traffic optimiser configurations.

Resolution

This issue is resolved in ESXi 7.0 U1 and later available at Broadcom Downloads.
This issue is resolved in ESXi 6.7 P05 and later available at Broadcom Downloads.
If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

Workaround:
Reconfigure the firewall, router or WAN traffic optimiser settings to filter out the timestamp option in both directions or not at all.

Additional Information

- https://tools.ietf.org/html/rfc7323 (3.2. Timestamps Option)
- pktcap-uw Command Syntax for Capturing Packets

Impact/Risks:
- Virtual Machine migrations across WANs are not successful