Starting VirtualCenter Server service on VMware vCenter Server Appliance (VCSA) fails with the error: vim.fault.AlreadyExists - Entry matching VCSA_FQDN:8089 already exists

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• The VMware VirtualCenter Server service fails to start on VMware vCenter Server Appliance (VCSA).
• Cannot start VirtualCenter Server service on VCSA.
• In the vpxd.log file, you see entries similar to:
  
  YYYY-MM-DDT<time>Z [7FAD42D91740 info 'vpxdvpxdMoOptionManager'] [OptionManagerMo] No change to vpxd.usageStats.duration
  YYYY-MM-DDT<time>Z [7FAD42D91740 info 'vpxdvpxdMoCbrcManager'] [CbrcManagerMo::Init]
  YYYY-MM-DDT<time>Z [7FAD42D91740 verbose 'vpxdvpxdNfcServiceProxy'] VpxdNfcServiceProxy::VpxdNfcServiceProxy
  YYYY-MM-DDT<time>Z [7FAD42D91740 verbose 'vpxdvpxdMoReverseProxy'] [VpxdReverseProxy] Proxy config path: /etc/vmware-vpx/proxy.xml
  YYYY-MM-DDT<time>Z [7FAD42D91740 info 'ProxySvc'] Max proxy connections currently set to 2048 , buffer size: 16344 bytes
  YYYY-MM-DDT<time>Z [7FAD42D91740 info 'ProxySvc'] vmacore/ssl/useSSLCtxPool: true
  YYYY-MM-DDT<time>Z [7FAD42D91740 info 'ProxySvc'] vmacore/ssl/serializeServerHandshake: false
  YYYY-MM-DDT<time>Z [7FAD42D91740 verbose 'vpxdvpxdMoReverseProxy'] [VpxdReverseProxy] Created endpoint httpsPort at 443
  YYYY-MM-DDT<time>Z [7FAD42D91740 info 'ProxySvc'] Max proxy connections currently set to 2048 , buffer size: 16344 bytes
  YYYY-MM-DDT<time>Z [7FAD42D91740 verbose 'vpxdvpxdMoReverseProxy'] [VpxdReverseProxy] Created endpoint httpPort at 80
  YYYY-MM-DDT<time>Z [7FAD42D91740 warning 'ProxySvc'] Entry matching VCSA_FQDN:8089/ already exists.</time>
  YYYY-MM-DDT<time>Z [7FAD42D91740 error 'vpxdvpxdMain'] [VpxdMain] Failed to initialize: vim.fault.AlreadyExists
  YYYY-MM-DDT<time>Z [7FAD42D91740 error 'Default'] Failed to intialize VMware VirtualCenter. Shutting down...
  YYYY-MM-DDT<time>Z [7FAD42D91740 info 'vpxdvpxdSupportManager'] Wrote uptime information</time></time></time></time></time></time></time></time></time></time></time></time></time>

Cause

This issue occurs due to a duplicate entry referencing port 8089 within the /etc/vmware-vpx/proxy.xml file.

Resolution

To resolve this issue, remove the duplicate entry referencing port 8089 within the /etc/vmware-vpx/proxy.xml file.

To remove the duplicate entry:

1. Log in to the vCenter Server Appliance as the root user.
2. Open the console.
3. Run this command:
   
   cd /etc/vmware-vpx
   
   
4. Run this command to view an output that contain all lines with 8089:
   
   grep "8089</serverNamespace>" proxy.xml
   
   You see output similar to:
   
   <serverNamespace>192.168.2.2:8089</serverNamespace>
   <serverNamespace>VC-DSN:8089</serverNamespace>
   <serverNamespace>[fe80::250:56ff:fe03:3f2]:8089</serverNamespace>
   <serverNamespace>sdkTunnel:8089</serverNamespace>
   
   
5. If you receive two similar lines from the output in step 4, make a copy of the proxy.xml file by running this command:
   
   cp /etc/vmware-vpx/proxy.xml /etc/vmware-vpx/proxy.bak
   
   
6. Open /etc/vmware-vpx/proxy.xml file using a text editor and locate the duplicated entries.
   
   <e id="26">
   <_type>vim.ProxyService.LocalTunnelSpec</_type>
   <accessMode>httpOnly</accessMode>
   <port>8089</port>
   <serverNamespace>vCenter.domain:8089</serverNamespace>
   </e>
   
   <e id="27">
   <_type>vim.ProxyService.LocalTunnelSpec</_type>
   <accessMode>httpOnly</accessMode>
   <port>8089</port>
   <serverNamespace>vCenter.domain:8089</serverNamespace>
   </e>
   
   
7. Remove the complete element with highest element ID (For example, 27). The endpoint entry looks similar to:
   
   <e id="27">
   <_type>vim.ProxyService.LocalTunnelSpec</_type>
   <accessMode>httpOnly</accessMode>
   <port>8089</port>
   <serverNamespace>vCenter.domain:8089</serverNamespace>
   </e>
   
   
8. The elementID values must be sequential - if after removing the duplicate element ID there is a gap in the sequence it is necessary to renumber the element ID values after the removed entry. For example, if element ID 27 was removed and there is an element ID of 28 then change this element ID from 28 to 27 and so on.
   
9. Locate this entry in the proxy.xml file and change the length variable to the total number of remaining endpoint IDs. This should be one less than the current variable
   <config>
   <EndpointList>
   <_length>29</_length>
   <_type>vim.ProxyService.EndpointSpec[]</_type>
   <e id="0">
   
   
10. Save the proxy.xml file and restart the VirtualCenter Server service.

Additional Information

The elementID values must be sequential (0,1,2,3,4,5...) - if there is a missing element ID in the sequence (0,1,2,4,5...) then vpxd fails to start with InvalidArgument in the vpxd log file.