ESXi domain join fails when SMBv1 is disabled on the domain controller or a firewall is blocking SMB negotiate packets

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:
An ESXi host may fail to join an Active Directory domain when the SMBv1 protocol is disabled on the domain controller or when a firewall is blocking SMB negotiate packets.

Environment

VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.5

Cause

SMB2 is supported on ESXi 6.5 Update 1 onward but the initial SMB packet negotiation request begins over SMB1 packet. If SMB2 is enabled on both Active Directory controller and the ESXi host then the negotiation switches to SMB2 otherwise it negotiates through SMB1packets only.

If SMB1 is disabled on the domain controller then it would prevent the initial packet negotiation, thus causing packet drops and eventually failure with an error similar to ERROR_GEN_FAILURE.

A firewall that is configured to block SMB negotiate packets may also cause this issue but will send a RST packet. To confirm, perform a packet capture during join:
 

# tcpdump-uw -n -i vmk0 -w join.pcap 

Use a wireshark filter 'tcp.port==445' to observe the SMB negotiate protocol packet and RST from the destination:

• 128 Negotiate Protocol Request
  • Requested Dialects
    NT LM 0.12
    SMB 2.002
• 60 445->22553 [RST] Seq=1 Win=1 Len=1

Note, numerous attempts to re-transmit indicates that the destination is not acknowledging or responding to the SMB negotiate protocol packet.

Resolution

This issue is resolved in ESXi 6.7 Update 2 where the initial packet negotiation will begin via SMB2 by default instead of SMB1, thus disabling SMB1 completely.

Workaround:
Disable any firewall rules blocking SMB negotiate packet. Note, ensure to enable firewall rules on esxi and firewalls between esxi host and domain controllers. 

For more information on ports that need to be opened, see "Errors in Active Directory operations" error adding the ESX/ESXi host to an Active Directory domain (1026538)"