Patching from vCenter 7.x to 7.0 U3o Fails Upon Starting VPXD Service
search cancel

Patching from vCenter 7.x to 7.0 U3o Fails Upon Starting VPXD Service

book

Article ID: 319005

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Help patching vCenter.

Symptoms:
-Patching from vCenter version 7.x to 7.0 U3o failed upon starting vpxd service.


in /var/log/vmware/applmgmt/PatchRunner.log you will find the below logs:

 

YYYY-MM-DDTHH:MM:SS INFO service_manager Service vmware-vpxd reported status stopped. Expected status started
YYYY-MM-DDTHH:MM:SS ERROR service_manager Service cannot be started. Error: Error executing start on service vpxd. Details {
     "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "vpxd"
            ],
            "localized": "An error occurred while starting service 'vpxd'"
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
}
Service-control failed. Error: {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "vpxd"
            ],
            "localized": "An error occurred while starting service 'vpxd'"
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
}   
YYYY-MM-DDTHH:MM:SS ERROR vmware_b2b.patching.phases.patcher Patch hook Patch got unhandled exception.
Traceback (most recent call last):
  File "/storage/core/software-update6ezrxg3x/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 203, in patch
    _patchComponents(ctx, userData, statusAggregator.reportingQueue)
  File "/storage/core/software-update6ezrxg3x/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 84, in _patchComponents
    _startDependentServices(c)
  File "/storage/core/software-update6ezrxg3x/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 53, in _startDependentServices
    serviceManager.start(depService)
  File "/storage/core/software-update6ezrxg3x/stage/scripts/patches/libs/sdk/service_manager.py", line 901, in wrapper
    return getattr(controller, attr)(*args, **kwargs)
  File "/storage/core/software-update6ezrxg3x/stage/scripts/patches/libs/sdk/service_manager.py", line 794, in start
    super(VMwareServiceController, self).start(serviceName)
  File "/storage/core/software-update6ezrxg3x/stage/scripts/patches/libs/sdk/service_manager.py", line 665, in start
    raise IllegalServiceOperation(errorText)
service_manager.IllegalServiceOperation: Service cannot be started. Error: Error executing start on service vpxd. Details {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "vpxd"
            ],
            "localized": "An error occurred while starting service 'vpxd'"
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
}
Service-control failed. Error: {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "vpxd"
            ],
            "localized": "An error occurred while starting service 'vpxd'"
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
}"




In /var/log/vmware/vmon/vmon.log, you will find the below logs:
 

YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127 Stdout = Status : Failed
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127 Error Code : 70012
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127 Error Message : Invalid CSR field
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127 
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127 Stderr = 
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127 
YYYY-MM-DDTHH:MM:SS Wa(03) host-25127 <vpxd> Service pre-start command's stderr: Traceback (most recent call last):
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127   File "/usr/lib/vmware-vpx/py/vpxd-prestart.py", line 68, in <module>
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127     patch_vpxd_prop()
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127   File "/usr/lib/vmware-vpx/py/vpxd_update.py", line 314, in patch_vpxd_prop
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127     updateGoscSpecDecertInB2BOrNDU()
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127   File "/usr/lib/vmware-vpx/py/vpxd_update.py", line 271, in updateGoscSpecDecertInB2BOrNDU
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127     decert.create()
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127   File "/usr/lib/vmware-vpx/py/data_encipherment_cert_utils.py", line 183, in create
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127     self._gen_cert()
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127   File "/usr/lib/vmware-vpx/py/data_encipherment_cert_utils.py", line 147, in _gen_cert
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127     raise e
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127   File "/usr/lib/vmware-vpx/py/data_encipherment_cert_utils.py", line 135, in _gen_cert
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127     invoke_command(cmd)
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127   File "/usr/lib/vmware/site-packages/cis/utils.py", line 369, in invoke_command
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127     (cmd, stderr))
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127 cis.exceptions.InvokeCommandException: {
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127     "detail": [
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127         {
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127             "id": "install.ciscommon.command.errinvoke",
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127             "translatable": "An error occurred while invoking external command : '%(0)s'",
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127             "args": [
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127                 "Command: ['/usr/lib/vmware-vmca/bin/certool', '--server=vc-FQDN', '--genCIScert', '--dataencipherment', '--privkey=/etc/vmware-vpx/ssl/tmp-data-encipherment.key', '--cert=/etc/vmware-vpx/ssl/tmp-data-encipherment.crt', '--Name=data-encipherment', '--IPAddress=vc-FQDN']\nStderr: "
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127             ],
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127             "localized": "An error occurred while invoking external command : 'Command: ['/usr/lib/vmware-vmca/bin/certool', '--server=vc-FQDN', '--genCIScert', '--dataencipherment', '--privkey=/etc/vmware-vpx/ssl/tmp-data-encipherment.key', '--cert=/etc/vmware-vpx/ssl/tmp-data-encipherment.crt', '--Name=data-encipherment', '--IPAddress=vc-FQDN']\nStderr: '"
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127         },
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127         {
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127             "id": "upgrade.vpxd.cert.create.failed",
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127             "translatable": "Failed to create data encipherment cert with hostname/ip %(0)s.",
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127             "args": [
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127                 "vc-FQDN"
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127             ],
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127             "localized": "Failed to create data encipherment cert with hostname/ip vc-FQDN."
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127         }
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127     ],
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127     "componentKey": null,
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127     "problemId": null,
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127     "resolution": null
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127 }
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-25127 
YYYY-MM-DDTHH:MM:SS Er(02) host-25127 <vpxd> Service pre-start command failed with exit code 1."



Environment

VMware vCenter Server 7.0.3

Cause

  • Data encipherment certificate is already expired or about to expire in 1 year.

Resolution

Please ensure to have valid snapshot (Offline snapshots of all nodes within an Enhanced Linked Mode environment) and/or file-based backups complete before making any changes

 

NOTE: If during the patch process you have no more retries allowed, please restore VC/s to backup/s and apply resolution steps and attempt patch again IF you have an option to retry, apply below steps and then select to retry patch: 

 

To resolve the issue we require to update the data-encipherment certificate to ensure all entries are correct: 

Manual Method to replace data-encipherment Certificate (appliance)

  • Take backup of old certificate and private key:

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store data-encipherment --alias data-encipherment --output /tmp/old-data-encipherment.crt

/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store data-encipherment --alias data-encipherment --output /tmp/old-data-encipherment.key

 

  • Delete the existing certificate from the VECS store:

/usr/lib/vmware-vmafd/bin/vecs-cli entry delete -y --store data-encipherment --alias data-encipherment

 

  • List the VECS store and confirm the list is Empty:

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store data-encipherment

 

  • Generate a new certificate using the existing private key, and add it to the VECS store
    Note: --genCIScert switch in certool will automatically add the new certificate to the Certificate Store

/usr/lib/vmware-vmca/bin/certool --server=localhost --genCIScert --dataencipherment --privkey=/tmp/old-data-encipherment.key --cert=/tmp/tmp-data-encipherment.crt --Name=data-encipherment --FQDN=<VC_FQDN>

 

  • Restart vpxd services: 

Service-control --stop vpxd

Service-control --start vpxd

 

  • Retry patch again

 

For additional information on above steps please see following: How to replace an expired data-encipherment certificate on vCenter Server

Additional Information

In PR 3292323 , it is advised that the issue is happening when patching to 8.0 U2 as the patch script has some code to replace the data-encipherment cert if about to expire but it seems that the issue is also happening when patching to version 7.0 U3o.

Impact/Risks:
  • Patching fails at starting vpxd service.


Powered by Wolken Software