vCenter Patch Failure at 80%-Exception occured in postInstallHook
search cancel

vCenter Patch Failure at 80%-Exception occured in postInstallHook

book

Article ID: 407923

calendar_today

Updated On:

Products

VMware vCenter Server VMware vCenter Server 8.0

Issue/Introduction

  • During a vCenter Server 7.x/8.x patch, the process fails at 80% with the following error:

  • From /var/log/vmware/appliance/patchRunner.log, the patch process shows failure when starting the vpxd-svcs service:

    YYYY-MM-DDThh:mm:ssZ ERROR vmware_b2b.patching.phases.patcher Patch hook Patch got unhandled exception.
Traceback (most recent call last):
  File "/storage/seat/software-update_blukhxe/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 208, in patch
    _patchComponents(ctx, userData, statusAggregator.reportingQueue)
  File "/storage/seat/software-update_blukhxe/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 89, in _patchComponents
    _startDependentServices(c)
  File "/storage/seat/software-update_blukhxe/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 56, in _startDependentServices
    serviceManager.start(depService)
  File "/storage/seat/software-update_blukhxe/stage/scripts/patches/libs/sdk/service_manager.py", line 909, in wrapper
    return getattr(controller, attr)(*args, **kwargs)
  File "/storage/seat/software-update_blukhxe/stage/scripts/patches/libs/sdk/service_manager.py", line 799, in start
    super(VMwareServiceController, self).start(serviceName)
  File "/storage/seat/software-update_blukhxe/stage/scripts/patches/libs/sdk/service_manager.py", line 665, in start
    raise IllegalServiceOperation(errorText)
service_manager.IllegalServiceOperation: Service cannot be started. Error: Error executing start on service vpxd-svcs. Details {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "vpxd-svcs"
            ],
            "localized": "An error occurred while starting service 'vpxd-svcs'"
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
}
Service-control failed. Error: {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "vpxd-svcs"
            ],
            "localized": "An error occurred while starting service 'vpxd-svcs'"
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
}
YYYY-MM-DDThh:mm:ssZ WARNING root stopping status aggregation...
YYYY-MM-DDThh:mm:ssZERROR __main__ Patch vCSA failed

     

  • From /var/log/vmware/vpxd-svcs/pre-start-vpxd-svcs.log, we see entries related to "Invalid certificate'

    YYYY-MM-DDThh:mm:ssZ ERROR:tagging_grpc_registration:Failed to reregister Tagging service grpc endpoints with Lookup Service
YYYY-MM-DDThh:mm:ssZ ERROR:tagging_grpc_registration:(vmodl.fault.InvalidArgument) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   msg = '',
   faultCause = <unset>,
   faultMessage = (vmodl.LocalizableMessage) [],
   invalidProperty = 'Invalid certificate'

     

  • From /var/log/vmware/vmon/vmon.log will show similar entries to:

    File "/usr/lib/python3.7/ssl.py", line 1139, in do_handshake
  self._sslobj.do_handshake()
ssl.SSLCertVerificationErrorL [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1076)
<vpxd-svcs> Service pre-start command failed with exit code 1.
  • While "vCert script " option 1 output will demonstrate the "MISSING CA OR INVALID OR EXPIRED" certificate:
    Checking Certificate Status
    -----------------------------------------------------------------
    Checking Machine SSL certificate                            VALID
    Checking Solution User certificates:
       machine                                             MISSING CA OR INVALID OR EXPIRED 
       vsphere-webclient                                   MISSING CA OR INVALID OR EXPIRED 
       vpxd                                                MISSING CA OR INVALID OR EXPIRED 
       vpxd-extension                                      MISSING CA OR INVALID OR EXPIRED 
       hvc                                                      VALID
       wcp                                                      VALID
    Checking SMS self-signed certificate                        VALID
    Checking SMS VMCA-signed certificate                        VALID
    Checking data-encipherment certificate                      VALID
    Checking Authentication Proxy certificate                   VALID
    Checking Auto Deploy CA certificate                       NO SKID
    Checking VMCA certificate                                   VALID

Environment

vCenter Appliance 7.x

vCenter Appliance 8.x

Cause

The issue occurs because one or more solution user certificates in vCenter are expired, invalid, or missing CA.

Resolution

NOTE:-Ensure backups or offline snapshots exist before making certificate changes. See: VMware vCenter in Enhanced Linked Mode – backup considerations

  • Revert to snapshot or backup taken in working state before the failed upgrade attempt.
  • Use the new VMware certificate management tool for all replacement workflows: VMware Certificate Management Tool – KB 385107

  • If solution user certificates are flagged as MISSING CA, INVALID, or EXPIRED, follow:

    • Option #3 → Manage certificates

    • Option #2 → Solution User certificates (This replaces the Solution User certificates in VECS and updates the Service Principal entries in VMware Directory.)

  • Once certificates are replaced and validated, re-run the patch process.

 

Powered by Wolken Software