vCLS VM keeps getting deployed and the task gets canceled by user 'com.vmware.vim.eam' and the vCLS VM gets deleted because of DNS issue.

Products

• VMware vCenter Server

Issue/Introduction

• vCLS VMs keep getting deployed and then shows that "The task was canceled by a user" and this user com.vmware.vim.eam then the vCLS VM gets deleted and recreated.
  
  
• Check for similar logs in vCenter at:/var/log/vmware/envoy

 2022-05-11T19:01:09.516Z info envoy[140192810850112] [Originator@6876 sub=Default] 2022-05-11T18:59:08.762Z CONNECT {HOSTNAME] :8089 HTTP/1.1 200 downstream_remote_disconnect DC 24634 71188 120042 - - 127.0.0.1:36376 127.0.0.1:80 127.0.0.1:59158 127.0.0.1:8089
2022-05-11T19:02:09.519Z info envoy[140192810850112] [Originator@6876 sub=Default] 2022-05-11T18:57:19.747Z CONNECT {HOSTNAME]:8089 HTTP/1.1 200 downstream_remote_disconnect DC 427564 802986 289030 - - 127.0.0.1:35872 127.0.0.1:80 127.0.0.1:58654 127.0.0.1:8089
2022-05-11T19:03:09.518Z info envoy[140192810850112] [Originator@6876 sub=Default] 2022-05-11T18:57:18.945Z CONNECT {HOSTNAME]:8089 HTTP/1.1 200 downstream_remote_disconnect DC 45660 130517 349898 - - 127.0.0.1:35864 127.0.0.1:80 127.0.0.1:58646 127.0.0.1:8089


• It takes 427564 MS  to have internal timeout for DNS resolution.
  
• In some cases you might find in the vCenter: /var/log/vmware/eam/eam.log file a similar error:
  
  JOB FAILED: Can't provision VM for ClusterAgent(ID: 'XXXX') due to lack of suitable datastore.

Cause

These errors are related to external DNS / 3rd party DNS issues in the environment.

Resolution

Use the following steps to identify where the issue resides within DNS:

1. If the issue is 100% reproducible then it could be a wrong DNS name.
   • Try to ping the host found in the logs above using the following command:
     
     ​​​​​​​ping <host_FQDN>


   • Try to nslookup / dig the ESX Host:
     ​​​​​​​
     ​​​​​​​nslookup <host_FQDN>
dig <host FQDN>


   • If the connection times out the issue is in the DNS record.
     
     
2. If the issue is not 100% reproducible, there may be a slow DNS.
   ​​​​​​​
   • Check the DNS server the vCenter is connected to, ping it, and check the time in millisecond:
     
     cat /etc/systemd/resolved.conf