vCenter Server Appliance 6.7 Upgrade Fails During Firstboot - VMware vSphere Profile-Driven Storage Service Firstboot Failed - "VMware vSphere Profile-Driven Storage Service failed to start" - "VPXD client login failed"

Products

• VMware vCenter Server

Issue/Introduction

To learn more about firstboot issues see: Firstboot Failed during Install, Deployment, Upgrade, or Migration in vCenter Server and vCenter Server Appliance 6.7
To collect a log bundle or review log files for a vCenter Server Appliance install, upgrade, or migration issues reference Collecting Logs for Troubleshooting a vCenter Server Installation or Upgrade The relevant logs will be on the newly deployed appliance.  

firstbootStatus.json contains the following: "failedSteps": "sms_spbm_firstboot"

sms_spbm_firstboot.py_####_stderr.log contains the following: VMware vSphere Profile-Driven Storage Service failed to start

fbInstall.json shows an unexpected time discrepancy between the start time and end time.  This could be a large jump forward or backward in time.  For example:

"start_time": "YYYY-MM-DDT13:00:00.000Z
"end_time": "YYYY-MM-DDT18:00:00.000Z"

or 

"start_time": "YYYY-MM-DDT18:05:00.000Z
"end_time": "YYYY-MM-DDT18:00:00.000Z"

In the first example, although the upgrade may have failed in twenty minutes, the logs indicate a five hour difference between the start and end time.  In the second example, the end time is a five minutes earlier than the start time which indicates a backward jump in time. 

Note: 

• vCenter Server Appliance - Firstboot logs are located in the /var/log/firstboot directory.

sps.log contains the following:

VPXD client login failed
 

Note:

• vCenter Server Appliance - Each service will have it's own folder in the /var/log/vmware/ directory.  The sps logs are located in the /var/log/vmware/vmware-sps folder.

Environment

VMware vCenter Server Appliance 6.7

Cause

This issue occurs due to time inconsistencies in the vSphere environment.  The issue most commonly happens when the target ESXi host for the destination vCenter Server Appliance is not synchronized with NTP.  This issue can also happen if the destination vCenter Server Appliance migrates to an ESXi host with different time due to fully automated DRS.

Resolution

To avoid time synchronization issues, ensure the following is correct before deploying, migrating, or upgrading a vCenter Server Appliance:

1. The target ESXi host where the destination vCenter Server Appliance will be deployed is synchronized to NTP.
2. If the target ESXi host is part of a Fully Automated DRS cluster, change the automation level to Manual.
3. The ESXi host running the source vCenter Server Appliance is synchronized to NTP.
4. If the vCenter Server Appliance will be connected to an external Platform Services Controller, ensure the ESXi host running the external Platform Services Controller is synchronized to NTP.
5. Verify that the source vCenter Server or vCenter Server Appliance and external Platform Services Controller have the correct time.

Additional Information

• For more information on managing time in vSphere, see Synchronizing Clocks on the vSphere Network.
• For more information on changing DRS Automation, see Create a vSphere DRS Cluster