vSphere with Tanzu/vSphere Kubernetes Supervisor Unable to resolve host with the ".local" suffix

Products

VMware vCenter Server 7.0 VMware vCenter Server 8.0 VMware vSphere 7.0 with Tanzu vSphere with Tanzu VMware vSphere with Tanzu Tanzu Kubernetes Runtime

Issue/Introduction

In the vSphere web client under Workload Management, the Supervisor cluster is in Error state with an error message similar to the following where my-hostname.local is the name of a host using the .local domain and can vary by environment:

Unable to resolve the host my-hostname.local on control plane VM 1234a56c78b9c10def1hijk123lm4n5o6p789. The hostname ends with the '.local' top level domain, which requires 'local' to be included in the management DNS search domains.

While SSH directly connected to any Supervisor Control Plane VM as root:

Ping, Dig and nslookup fail to any hostnames ending in .local. In the below example, my-hostname.local is used.

ping: my-hostname.local: Temporary failure in name resolution

The correct DNS server is in use:
- ```
resolvectl status
```
The correct DNS server is reachable.

Environment

vSphere with Tanzu 7.0

vSphere with Tanzu 8.0

This issue can occur regardless of whether the environment is managed by Tanzu Mission Control (TMC) or not.

Cause

This is not unique to vSphere with Tanzu or vSphere Kubernetes Supervisor. This is expected behavior from the systemd-resolved service.

https://www.freedesktop.org/software/systemd/man/systemd-resolved.service.html

Multi-label names with the domain suffix ".local" are resolved using MulticastDNS on all local interfaces where MulticastDNS is enabled. As with LLMNR, IPv4 address lookups are sent via IPv4 and IPv6 address lookups are sent via IPv6.
Queries for multi-label names are routed via unicast DNS on local interfaces that have a DNS server configured, plus the globally configured DNS servers if there are any. Which interfaces are used is determined by the routing logic based on search and route-only domains, described below. Note that by default, lookups for domains with the ".local" suffix are not routed to DNS servers, unless the domain is specified explicitly as routing or search domain for the DNS server and interface. This means that on networks where the ".local" domain is defined in a site-specific DNS server, explicit search or routing domains need to be configured to make lookups work within this DNS domain. Note that these days, it's generally recommended to avoid defining ".local" in a DNS server, as RFC6762 reserves this domain for exclusive MulticastDNS use.

Due to the above, VMware by Broadcom does not recommend any deployment which uses .local for any component, including vCenter, ESXi, NSX Manager, NSX Edge Nodes and/or any endpoint used by vSphere Kubernetes clusters.

Resolution

Note: If .local must be used, it is advised to only be used for Proof of Concept (POC) and Lab environments.

Please see the below for a workaround in using .local in a non-production environment.

Overview:

Ensure that the master_DNS_search domains includes the corresponding .local domain for your environment.
Confirm that for each Supervisor control plane VM, the /etc/hosts file has the vCenter FQDN with the .local domain and its IP address.
Check that resolvectl is set appropriately for the .local domain within the vSphere Kubernetes cluster nodes.

Master_DNS_Search Domains

While connected to the VCSA as root, retrieve the cluster domain name. In the below example, this is domain-c8:

root@vcsa-01a [ ~ ]# dcli com vmware vcenter namespacemanagement clusters list
|---------|------------|---------------------------------------------------------------------------------|-----------------|-------------|
|cluster  |cluster_name|stats                                                                            |kubernetes_status|config_status|
|---------|------------|---------------------------------------------------------------------------------|-----------------|-------------|
|domain-c8|            ||--------|----------------|-----------|------------|---------------|------------||WARNING          |RUNNING      |
|         |            ||cpu_used|storage_capacity|memory_used|cpu_capacity|memory_capacity|storage_used||                 |             |
|         |            ||--------|----------------|-----------|------------|---------------|------------||                 |             |
|         |            ||0       |0               |0          |0           |0              |2048        ||                 |             |
|         |            ||--------|----------------|-----------|------------|---------------|------------||                 |             |
|---------|------------|---------------------------------------------------------------------------------|-----------------|-------------|
root@vcsa-01a [ ~ ]#

Perform the following command to check the current search domain with the cluster domain name output from the previous step. In the below example, this is domain-c8:

root@vcsa-01a [ ~ ]# dcli com vmware vcenter namespacemanagement clusters get --cluster domain-c8 | grep -A 1 master_DNS_search_domains
master_DNS_search_domains:
   - corp.tanzu
root@vcsa-01a [ ~ ]#

If the .local search domain is not present, perform the below update command to add the intended .local search domain. In the below example, this is test.local.
```
root@vcsa-01a [ ~ ]# dcli com vmware vcenter namespacemanagement clusters update --cluster domain-c8 --master-dns-search-domains test.local
```

Confirm that the .local domain was added successfully:

root@vcsa-01a [ ~ ]# dcli com vmware vcenter namespacemanagement clusters get --cluster domain-c8 | grep -A 1 master_DNS_search_domains
master_DNS_search_domains:
   - test.local

Supervisor Control Plane VM /etc/hosts

For each Supervisor Control Plane VM, ssh into the Supervisor Control Plane VM as root:
- See "How to SSH into Supervisor Control Plane VMs" in KB: Troubleshooting vSphere with Tanzu (TKGS) Supervisor Control Plane VMs
Edit the /etc/hosts file to include the vCenter FQDN and IP address. For example:
- ```
10.1.12.34 my-center-host.local
```

vSphere Kubernetes Cluster resolvectl

On each node in the vSphere Kubernetes Cluster, run the below commands according to your .local domain. In the below example, this is testds.local:
- ```
resolvectl status
```
- ```
resolvectl domain eth0 testds.local
```
- Note: The above value will persist through reboots but will need to be manually applied on each new upgraded node after a TKR upgrade.

Additional Information

Impact/Risks:
The .local hostname is reserved for use in mDNS per RFC6762 therefore trying to resolve it against a DNS server violates RFC6762. As such VMware does not recommend any deployment which uses .local for any components. (this includes vCenter, esxi, nsx manager, nsx edge nodes, and any endpoint TKGS uses like harbor).

The workaround for this is STRICTLY for proof of concept and lab use. Implementing this workaround in your production environment could result in unexpected scenarios as our Quality Assurance Team has not validated/tested this configuration and does not plan to validate in the future.