"[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:720)" while upgrading to VCSA 6.7

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:
To learn more about firstboot issues see: Understanding and Troubleshooting vCenter Server and vCenter Server Appliance 6.7 Firstboot Install/Deployment, Upgrade, or Migration Failures​.
To collect a log bundle or review log files for a vCenter Server Appliance install, upgrade, or migration issues reference Triaging a vCenter Server Appliance 6.0 installation, upgrade, or migration (6.0 U2m) failure.  The relevant logs will be on the newly deployed appliance.  

vCenter Server Appliance 6.7 Upgrade Fails During Firstboot - VMware vCenter Server Firstboot Failed - "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:720)"

firstbootStatus.json contains the following:

 "failedSteps": "vpxd_firstboot"

 
vpxd_firstboot.py_####_stdout.log contains the following:

ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:720)

fbInstall.json shows an unexpected time discrepancy between the start time and end time.  This could be a large jump forward or backward in time.  For example:

 

"start_time": "2018-05-07T13:00:00.000Z
"end_time": "2018-05-07T18:00:00.000Z"

 

or

"start_time": "2018-05-07T18:05:00.000Z
"end_time": "2018-05-07T18:00:00.000Z"

In the first example, although the upgrade may have failed in twenty minutes, the logs indicate a five hour difference between the start and end time.  In the second example, the end time is a five minutes earlier than the start time which indicates a backward jump in time. 
 
Note: 

• vCenter Server Appliance - Firstboot logs are located in the /var/log/firstboot directory.

Environment

VMware vCenter Server Appliance 6.7.x

Cause

This issue occurs due to time inconsistencies in the vSphere environment.  The issue most commonly happens when the target ESXi host for the destination vCenter Server Appliance is not synchronized with NTP.  This issue can also happen if the destination vCenter Server Appliance migrates to an ESXi host with different time due to fully automated DRS.

Resolution

To avoid time synchronization issues, ensure the following is correct before deploying, migrating, or upgrading a vCenter Server Appliance:

1. The target ESXi host where the destination vCenter Server Appliance will be deployed is synchronized to NTP.
2. If the target ESXi host is part of a Fully Automated DRS cluster, change the automation level to Manual.
3. The ESXi host running the source vCenter Server Appliance is synchronized to NTP.
4. If the vCenter Server Appliance will be connected to an external Platform Services Controller, ensure the ESXi host running the external Platform Services Controller is synchronized to NTP.
5. Verify that the source vCenter Server or vCenter Server Appliance and external Platform Services Controller have the correct time.

For more information on managing time in vSphere, see Synchronizing Clocks on the vSphere Network.
For more information on changing DRS Automation, see Edit Cluster Settings.
For more information on vCenter Server Appliance requirements, see System Requirements for the vCenter Server Appliance and Platform Services Controller Appliance.