Firstboot fails when upgrading vCenter Server Appliance 6.7 and vCenter Server 7.x due to time synchronization issue
search cancel

Firstboot fails when upgrading vCenter Server Appliance 6.7 and vCenter Server 7.x due to time synchronization issue

book

Article ID: 321371

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • The components of the vSphere environment are not time synchronized.
  • Firstboot fails during Install/Deployment, Upgrade or Migration.
  • In the firstbootStatus.json file, one of these services may have failed to configure or start during firstboot:
    • "failedSteps": "cmfirstboot"
    • "failedSteps": "analytics_firstboot"
    • "failedSteps": "vpxd_firstboot"
    • "failedSteps": "pschealth-firstboot”
    • "failedSteps": "sms_spbm_firstboot"
    • "failedSteps": "vmafd-firstboot"
    • "failedSteps": "vapi_firstboot"
    • "failedSteps": "mgmt-firstboot"
    • "failedSteps": "scafirstboot"
    • "failedSteps": "updatemgr-firstboot"
    • "failedSteps": "ngc_firstboot"

  • In the cmfirstboot.py_####_stderr.log file:

    PAM: Authentication token is no longer valid

    An error occurred while performing security operation: 'Failed to add user: cm to group: cis’

  • In the analytics_firstboot.py_####_stderr.log file:

    Analytics Service registration with Component Manager failed
    ns0:MessageExpired
    The time now (date + time) does not fall in the request lifetime interval extended with clock tolerance of 600000 ms [ (date + time); (date +time)]. This might be due to a clock skew problem.

  • In the vpxd_firstboot.py_####_stdout.log file:

    ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:720)

  • In the pschealth-firstboot.py_#####_stderr.log file:

    An error occurred while starting service 'pschealth'

  • In the sms_spbm_firstboot.py_####_stderr.log file:

    VMware vSphere Profile-Driven Storage Service failed to start

  • In the vmafdd-syslog.log file:

    Vmdir server is down.

  • In the vmdird-syslog.log file:
DecodeEntry failed (9605) DN:()
LoadServerGlobals: (9700)()
  • In the vapi_firstboot.py_####_stderr.log file:
Failed to configure vAPI Endpoint Service at the firstboot time
  • In the mgmt-firstboot.py_####_stderr.log file:
UnboundLocalError: local variable 'e' referenced before assignment
  • In the scafirstboot.py_####_stderr.log file:
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:720)
  • In the updatemgr-firstboot.py_6012_stderr.log file:

    Failed to register updatemgr extension

  • In the ngc_firstboot.py_####_stderr.log file:

    SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:720)

  • In the %ProgramData%\VMware\vCenterServer\logs\cm\cm.log file:
Caused by: com.vmware.vim.vmomi.client.exception.VlsiCertificateException: Server certificate chain is not trusted and thumbprint verification is not configured
at com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager.checkServerTrusted(ThumbprintTrustManager.java:183)
at sun.security.ssl.AbstractTrustManagerWrapper.checkServerTrusted(SSLContextImpl.java:984)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1496)
... 78 more
Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: timestamp check failed
  • In the fbInstall.json file shows an unexpected time discrepancy between the start time and end time. This could be a large jump forward or backward in time.

    For Example

    "start_time": "2018-05-07T13:00:00.000Z
    "end_time": "2018-05-07T18:00:00.000Z"

    or

    "start_time": "2018-05-07T18:05:00.000Z
    "end_time": "2018-05-07T18:00:00.000Z"

Environment

VMware vCenter Server Appliance 6.7.x
VMware vCenter Server 7.0.x

Cause

This issue occurs due to time inconsistencies in the vSphere environment. The issue most commonly happens when the target ESXi host for the destination vCenter Server Appliance is not synchronized with NTP. This issue can also happen if the destination vCenter Server Appliance migrates to an ESXi host with different time due to fully automated DRS.

Resolution

To avoid time synchronization issues, ensure the following is correct before deploying, migrating, or upgrading a vCenter Server Appliance:
  1. The target ESXi host where the destination vCenter Server Appliance will be deployed is synchronized to NTP.
  2. If the target ESXi host is part of a Fully Automated DRS cluster, change the automation level to Manual.
  3. The ESXi host running the source vCenter Server Appliance is synchronized to NTP.
  4. If the vCenter Server Appliance will be connected to an external Platform Services Controller, ensure the ESXi host running the external Platform Services Controller is synchronized to NTP.
  5. Verify that the source vCenter Server or vCenter Server Appliance and external Platform Services Controller have the correct time.
For more information on:

Additional Information

Note: Firstboot logs are located at:

  • vCenter Server Appliance - Firstboot logs are located in the /var/log/firstboot directory.
  • vCenter Server on Windows - Firstboot logs are located in the VMware-VCS-logs-.zip/vcs_logs/uninstall directory

    or

    VMware-VCS-logs-/vcs_logs/uninstall/.zip/ProgramData/VMware/vCenterServer/logs/firstboot directory
Note: In vSphere 7.0, vCenter Server for Windows has been removed and support is not available. For more information, see Farewell, vCenter Server for Windows .

For more information, see VMware vCenter Failed Firstboot.

-------------------------------------------------------------------------------------------------------------------------------------------------
For more information see:

Powered by Wolken Software