vpxd service fails to start after vCenter Server is migrated from vSphere 6.0 to 6.5

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• The vpxd service fails to start after vCenter Server is migrated from vSphere 6.0 to 6.5
• In the vpxd.log file, you see entries similar to:
  
  yyyy-mm-ddTxx:xx:xx.xxxZ error vpxd[7F2519F4A800] [Originator@6876 sub=AuthzStorageProvider] [AuthzStorageProvider::CreateAuthzMgr] Failed to connect to IS: <N5Vmomi5Fault17HostCommunication9ExceptionE(vmodl.fault.HostCommunication)<BR>--> [context]zKq7AVECAAAAAAomUQATdnB4ZAAADJQrbGlidm1hY29yZS5zbwAABE8bAGqXGAEOylR2cHhkAAF2E2IBsu71ATyJogFHdKICuBYBbGliYXV0aHpjbGllbnQuc28AAthIAQLwTgECytoBAkjQAQEs6KABCulUATjyVAFqlVMD4AUCbGliYy5zby42AAF1jVM=[/context]>
  yyyy-mm-ddTxx:xx:xx.xxxZ info vpxd[7F2519F4A800] [Originator@6876 sub=AuthzStorageProvider] [AuthzStorageProvider::CreateAuthzMgr] Retry for this error: attempt count 60
  yyyy-mm-ddTxx:xx:xx.xxxZ warning vpxd[7F2519F4A800] [Originator@6876 sub=VpxdAuthClient] [ConnectAndLogin] Failed to loginBySamlToken: N7Vmacore3Ssl18SSLVerifyExceptionE(SSL Exception: Verification parameters:
  --> PeerThumbprint: 07:E9:10:79:D5:12:1E:62:1A:41:20:72:0C:85:10:E9:19:8C:1F:05
  --> ExpectedThumbprint:
  --> ExpectedPeerName: localhost
  --> The remote host certificate has these problems:
  -->
  --> * Host name does not match the subject name(s) in certificate.
  -->
  --> * unable to get local issuer certificate)
   
• In the vpxd_firstboot.py_stderr.log file, you see entries similar to:
  
  yyyy-mm-ddTxx:xx:xx.xxxZ ERROR starting vpxd rc: 2, stdout: , stderr: Start service request failed. Error: Service crashed while starting
  yyyy-mm-ddTxx:xx:xx.xxxZ VirtualCenter firstboot failed
   
• In the cloudvm.log file, you see entries similar to:
  
  yyyy-mm-ddTxx:xx:xx.xxxZ: Failed to start services. Firstboot Error.
   
• In the fbInstall.json file, you see entries similar to:
  
  id": "install.ciscommon.service.failstart",
  "localized": "An error occurred while starting service 'vpxd'",
  "translatable": "An error occurred while starting service '%(0)s'"
   
• In the vmon-syslog.log file, you see entries similar to:
  
  17-04-25T03:16:43.069932+00:00 warning vmon Service vpxd exited. Exit code 1
   
• In the rhttpproxy.log file, you see entries similar to:
  
  yyyy-mm-ddTxx:xx:xx.xxxZ warning rhttpproxy[7F57DF421700] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f57b4038d40, h:27, <TCP '0.0.0.0:0'>, <TCP '127.0.0.1:8085'>>, e: system:111(Connection refused)
  yyyy-mm-ddTxx:xx:xx.xxxZ warning rhttpproxy[7F57DF421700] [Originator@6876 sub=Proxy Req 02531] Connection to localhost : 8085 failed with error N7Vmacore15SystemExceptionE(Connection refused)
  --> [context]zKq7AVECAAAAAAomUQAPcmh0dHBwcm94eQAADJQrbGlidm1hY29yZS5zbwAABE8bAGqXGABcVSsAGUwjAMpTJgC+VSYAGXQmAPp/JgBfFiQAL+IjABblIwD36ysBVHQAbGlicHRocmVhZC5zby4wAAItjg5saWJjLnNvLjYA[/context]

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Cause

This issue occurs because the vmdir rejects SSL connect due to hand shake failure. Although vpxd-svcs successfully connects to LDAP but looses connection later and then never re-connects.

Resolution

Note: Ensure to take a snapshot of vCenter Server and backup of vCenter Server Database before implementing below work around.

To work around this issue, use any one of these options:

• Re-generate certificates on existing vCenter server 6.0 and then start the migration to VCSA 6.5.
• Re-install vCenter server 6.0 using existing DB and then migrate to VCSA 6.5.

If this article didn’t solve your issue, submit your question to Broadcom Communities.

 

Additional Information

Resolution: "'PSC_FQDN' doesn't match with hostname in Platform Services Controller certificate" error installing vCenter Server 6.7