VM operations hang due to case-insensitive handling of vCenter Server FQDN by the Envoy sidecar proxy

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:
On vCenter Server 8.0U2, when attempting to perform VM operations, the operation hangs and completes after an unknown amount of time or fails to complete.

• Examples of affected VM operations:
  • Powering on
  • Rebooting
  • Cloning
  • Deployment of VMs via OVA/OVF

In /var/log/vmware/vsm/vsm.log, you may see:

2023-12-20T15:48:35.017+01:00 INFO [Thread-4] VCenterListener.java 485 - Connecting to https://VCENTER.domain.com:8089/sdk/vimService with vCenter proxy http://localhost:1080
...
2023-12-20T15:48:35.020+01:00 ERROR [Thread-4] VCenterListener.java 524 - com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 404
2023-12-20T15:48:35.020+01:00 ERROR [Thread-4] VCenterListener.java 284 - Unable to connect to the vCenter server. Retrying in 5 seconds
 

In /var/log/vmware/envoy/envoy-access.log, you may see capital letters in the vCenter's FQDN:

2023-12-19T16:44:35.803Z info envoy[2129] [Originator@6876 sub=Default] 2023-12-19T16:44:32.170Z CONNECT VCENTER.domain.com:8089 HTTP/1.1 404 route_not_found NR 0 0 0 - - 127.0.0.1:51414 - 127.0.0.1:1080 - - - -
2023-12-19T16:44:45.805Z info envoy[2129] [Originator@6876 sub=Default] 2023-12-19T16:44:37.306Z CONNECT VCENTER.domain.com:8089 HTTP/1.1 404 route_not_found NR 0 0 0 - - 127.0.0.1:51436 - 127.0.0.1:1080 - - - -
2023-12-19T16:44:45.805Z info envoy[2129] [Originator@6876 sub=Default] 2023-12-19T16:44:42.537Z CONNECT VCENTER.domain.com:8089 HTTP/1.1 404 route_not_found NR 0 0 0 - - 127.0.0.1:42670 - 127.0.0.1:1080 - - - -

In /var/log/vmware/rhttpproxy/rhttpproxy.log, you may see all lower case letters for the vCenter's FQDN:

2022-03-15T15:52:26.920Z info rhttpproxy[122452] [Originator@6876 sub=RhttpProxy] PNID: "vcenter.domain.com"

When checking vCenter's registered DCName, the FQDN contains capital letters.

root@VCENTER [ ~ ]# /opt/likewise/bin/lwregshell list_values "[HKEY_THIS_MACHINE\Services\vmafd\Parameters]" |grep 'DCName'
+  "DCName"                  REG_SZ          "VCENTER.domain.com"

Environment

VMware vCenter Server 8.0.2

Cause

Due to case-insensitive handing of the vCenter Server's FQDN in the destination URL by the Envoy sidecar proxy, when the vCenter Server hostnames are in upper case, calls to the VSM service hang.

Resolution

Engineering is aware of this issue and is actively working to resolve this issue.

Workaround:
As a workaround, update the vCenter Server's FQDN to lower case.
Note: Before proceeding, take a snapshot and/or perform a back up of the vCenter Server.

1. Export the vCenter Server likewise registry
   1. SSH to vCenter via root
   2. Stop all vCenter services: service-control --stop --all
   3. Start the likewise registry: systemctl start lwsmd
   4. Export the likewise registry to a text file: /opt/likewise/bin/lwregshell export /tmp/export.reg
2. Update the vCenter Server FQDN from UPPER CASE to all lower case
   1. Edit the exported likewise registry: vi /tmp/export.reg
   2. Update any references of the vCenter Server FQDN to all lower case
   3. Save the changes 
3. Import the newly edited registry: /opt/likewise/bin/lwregshell import /tmp/export.reg
4. Change the vCenter Server FQDN in the below files by running the following 3 commands:

sed -i -e 's/VCENTER.domain.com/vcenter.domain.com/g' /etc/vmware/systemname_info.json

sed -i -e 's/VCENTER.domain.com/vcenter.domain.com/g' /etc/vmware-rhttpproxy/endpoints.conf.d/vpxd-rhttpproxy-endpoint.conf

sed -i -e 's/VCENTER.domain.com/vcenter.domain.com/g' /etc/vmware-rhttpproxy/endpoints.conf.d/ProxyConfiguration.json

Note: In the above commands, adjust VCENTER.domain.com and vcenter.domain.com to reflect your vCenter Server instance. 

6. Reboot vCenter

Additional Information

Impact/Risks:
Depending on which workaround option applied, a restart of the vCenter Server services may be required.

Note: As a precaution, please take any necessary snapshots and/or perform a backup of the vCenter Server.