vCenter Server Appliance 6.7 Upgrade Fails During Firstboot - VMware Authentication Framework Firstboot Failed - "Vmdir server is down" - "(DecodeEntry failed (9605) DN:()"

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:
To learn more about firstboot issues see: Understanding and Troubleshooting vCenter Server and vCenter Server Appliance 6.7 Firstboot Install/Deployment, Upgrade, or Migration Failures​.

firstbootStatus.json contains the following:

 
"failedSteps": "vmafd-firstboot"

 
fbInstall.json shows an unexpected time discrepancy between the start time and end time.  This could be a large jump forward or backward in time.  For example:

"start_time": "2018-05-07T13:00:00.000Z
"end_time": "2018-05-07T18:00:00.000Z"

or 

"start_time": "2018-05-07T18:05:00.000Z
"end_time": "2018-05-07T18:00:00.000Z"

In the first example, although the upgrade may have failed in twenty minutes, the logs indicate a five hour difference between the start and end time.  In the second example, the end time is a five minutes earlier than the start time which indicates a backward jump in time. 
 
Note: 

• vCenter Server Appliance - Firstboot logs are located in the /var/log/firstboot directory.

vmafdd-syslog.log contains the following:

Vmdir server is down.

vmdird-syslog.log contains the following:

DecodeEntry failed (9605) DN:()
LoadServerGlobals: (9700)()
 

Note:

• vCenter Server Appliance - Each service will have it's own folder in the /var/log/vmware/ directory.  The vmafdd-syslog logs are located in the /var/log/vmware/vmafdd folder.  The vmdird-syslog logs are located in the /var/log/vmware/vmdird folder.

Environment

VMware vCenter Server Appliance 6.7.x

Cause

This issue occurs due to time inconsistencies in the vSphere environment.  The issue most commonly happens when the target ESXi host for the destination vCenter Server Appliance is not synchronized with NTP.  This issue can also happen if the destination vCenter Server Appliance migrates to an ESXi host with different time due to fully automated DRS.

Resolution

To avoid time synchronization issues, ensure the following is correct before deploying, migrating, or upgrading a vCenter Server Appliance:

1. The target ESXi host where the destination vCenter Server Appliance will be deployed is synchronized to NTP.
2. If the target ESXi host is part of a Fully Automated DRS cluster, change the automation level to Manual.
3. The ESXi host running the source vCenter Server Appliance is synchronized to NTP.
4. If the vCenter Server Appliance will be connected to an external Platform Services Controller, ensure the ESXi host running the external Platform Services Controller is synchronized to NTP.
5. Verify that the source vCenter Server or vCenter Server Appliance and external Platform Services Controller have the correct time.

For more information on managing time in vSphere, see Synchronizing Clocks on the vSphere Network.
For more information on changing DRS Automation, see Edit Cluster Settings.
For more information on vCenter Server Appliance requirements, see System Requirements for the vCenter Server Appliance and Platform Services Controller Appliance.