Adding ESXi host to AD domain using VMware Authentication proxy fails with error "The specified vSphere Authentication Proxy server is not reachable, or has denied access to the service"

Products

• VMware vSphere ESXi

Issue/Introduction

• When trying to add ESXi host to AD domain using VMware Authentication proxy fails with error

  The specified vSphere Authentication Proxy server is not reachable, or has denied access to the service

• We can add the ESXi host to the AD domain directly with the same user account with no issues.
• We see the following message in /var/log/hostd.log in ESXi

  YYYY-MM-DDTHH:MM:SS error hostd[2100851] [Originator@6876 sub=Default opID=lxn15o1q-98670-auto-244v-h5:70016208-44-57-641e user=vpxuser:domain_name\user_name] CamHttpJoinComputeNode: 13
  YYYY-MM-DDTHH:MM:SS error hostd[2100851] [Originator@6876 sub=ActiveDirectoryAuthentication opID=lxn15o1q-98670-auto-244v-h5:70016208-44-57-641e user=vpxuser:domain_name\user_name] vmwauth ConnectionRefusedException: N6vmware14authentication26ConnectionRefusedExceptionE(Exception 0x000004c9: The remote computer refused the network connection.)
  YYYY-MM-DDTHH:MM:SS info hostd[2100851] [Originator@6876 sub=Vimsvc.ha-eventmgr opID=lxn15o1q-98670-auto-244v-h5:70016208-44-57-641e user=vpxuser:domain_name\user_name] Event 8765 : Join domain failed.
  YYYY-MM-DDTHH:MM:SS info hostd[2100851] [Originator@6876 sub=Vimsvc.TaskManager opID=lxn15o1q-98670-auto-244v-h5:70016208-44-57-641e user=vpxuser:domain_name\user_name] Task Completed : haTask-ha-host-vim.host.ActiveDirectoryAuthentication.joinDomainWithCAM-1632422 Status error

• We see the following message in /var/log/vmware/vpxd/vpxd.log in vCenter

  YYYY-MM-DDTHH:MM:SS info vpxd[06940] [Originator@6876 sub=vpxLro opID=lxn15o1q-98670-auto-244v-h5:70016208-44] [VpxLRO] -- BEGIN task-182361 -- activeDirectoryAuthentication-94 -- vim.host.ActiveDirectoryAuthentication.joinDomainWithCAM -- 5255832a-43bd-330c-d116-ba37e3b06939(52959b06-e4ee-e4d8-1260-3c4f05013320)
  YYYY-MM-DDTHH:MM:SS info vpxd[06940] [Originator@6876 sub=vpxLro opID=lxn15o1q-98670-auto-244v-h5:70016208-44] [VpxLRO] -- FINISH task-182361
  YYYY-MM-DDTHH:MM:SS info vpxd[06940] [Originator@6876 sub=Default opID=lxn15o1q-98670-auto-244v-h5:70016208-44] [VpxLRO] -- ERROR task-182361 -- activeDirectoryAuthentication-94 -- vim.host.ActiveDirectoryAuthentication.joinDomainWithCAM: vim.fault.CAMServerRefusedConnection:
  --> Result:
  --> (vim.fault.CAMServerRefusedConnection) {
  -->    faultCause = (vmodl.MethodFault) null,
  -->    faultMessage = <unset>,
  -->    errorCode = 1225,
  -->    camServer = "vcenter_IP"
  -->    msg = "The specified vSphere Authentication Proxy server is not reachable, or has denied access to the service."
  --> }
  --> Args:
  -->
  --> Arg domainName:
  --> "domain_name"
  --> Arg camServer:
  --> "vcenter_IP"

• Checking the packet capture taken at the vCenter we can see the following

  663       YYYY-MM-DDTHH:MM:SS.646674    AD_DC_IP    VCSA_IP   KRB5    239    KRB Error: KRB5KDC_ERR_PREAUTH_REQUIRED
  1394    YYYY-MM-DDTHH:MM:SS.872140    AD_DC_IP   VCSA_IP   KRB5    369    KRB Error: KRB5KDC_ERR_S_PRINCIPAL_UNKNOWN

• Checking the same frame we see the following

  Frame 1378: 369 bytes on wire (2952 bits), 369 bytes captured (2952 bits)
  Ethernet II, Src: Mellanox_4b:4f:80 (MAC_Address), Dst: VMware_48:f2:5e (MAC_Address)
  Internet Protocol Version 4, Src: AD_DC_IP, Dst: VCSA_IP
  Transmission Control Protocol, Src Port: 88, Dst Port: 46536, Seq: 1, Ack: 1793, Len: 315
  Kerberos
      Record Mark: 311 bytes
      krb-error
          pvno: 5
          msg-type: krb-error (30)
          stime: YYYY-MM-DDTHH:MM:SS.000000000 Time zone
          susec: 121865
          error-code: eRR-S-PRINCIPAL-UNKNOWN (7)
          realm: domain_name
          sname
              name-type: kRB5-NT-SRV-HST (3)
              sname-string: 2 items
                  SNameString: ldap
                  SNameString: AD_DC_IP 

 

Environment

vCenter 7.0

Cause

• The SNameString in the above listed packet should have the FQDN of the domain controller in the customer environment.
• In case the value is IP address of the domain controller or IP address or hostname of the VCSA, you will see the error.
• This is caused due to DNS issues if the servers do not have both forward and reverse lookup entries configured on the Domain controller. 

Resolution

1. Verify the both forward and reverse entries exist for the domain controller

   nslookup domain_name

2. For each IP returned from the previous command, run `nslookup` i.e.:

   nslookup <IP-address-n>

3. Also, run the 'nslookup' commands from above explicitly against the DNS server configured.

   nslookup <search_entry> <dns_server_ip>

   This will help us isolate the issue.
   
   
4. Create DNS records accordingly.

Additional Information

Error "KRB5KDC_ERR_S_PRINCIPAL_UNKNOWN"