Error: "Failed to create data encipherment cert with hostname/ip" when patching vCenter

Products

• VMware vCenter Server

Issue/Introduction

• vCenter patching fails at 80%
  
  Error: Exception occurred in PostHookInstall during trying to start vpxd service.
  
  
• PatchRunner.log will display the following log messages indicating failure upon starting vpxd service:

vCenter - /var/log/vmware/applmgmt/PatchRunner.log

2023-10-28T04:37:18.555Z ERROR vmware_b2b.patching.phases.patcher Patch hook Patch got unhandled exception.
Traceback (most recent call last):
  File "/storage/updatemgr/software-updatew145uc9g/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 203, in patch
    _patchComponents(ctx, userData, statusAggregator.reportingQueue)
  File "/storage/updatemgr/software-updatew145uc9g/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 84, in _patchComponents
    _startDependentServices(c)
  File "/storage/updatemgr/software-updatew145uc9g/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 53, in _startDependentServices
    serviceManager.start(depService)
  File "/storage/updatemgr/software-updatew145uc9g/stage/scripts/patches/libs/sdk/service_manager.py", line 901, in wrapper
    return getattr(controller, attr)(*args, **kwargs)
  File "/storage/updatemgr/software-updatew145uc9g/stage/scripts/patches/libs/sdk/service_manager.py", line 794, in start
    super(VMwareServiceController, self).start(serviceName)
  File "/storage/updatemgr/software-updatew145uc9g/stage/scripts/patches/libs/sdk/service_manager.py", line 665, in start
    raise IllegalServiceOperation(errorText)
service_manager.IllegalServiceOperation: Service cannot be started. Error: Error executing start on service vpxd. Details {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "vpxd"
            ],
            "localized": "An error occurred while starting service 'vpxd'"
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
}
Service-control failed. Error: {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "vpxd"
            ],
            "localized": "An error occurred while starting service 'vpxd'"
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
}

• vCenter - /var/log/vmware/vmon/vmon.log

2023-10-28T04:37:17.122Z Wa(03)+ host-13732 2023-10-28T04:37:17.122Z  RC = 124
2023-10-28T04:37:17.122Z Wa(03)+ host-13732 Stdout = Status : Failed
2023-10-28T04:37:17.122Z Wa(03)+ host-13732 Error Code : 70012
2023-10-28T04:37:17.122Z Wa(03)+ host-13732 Error Message : Invalid CSR field
2023-10-28T04:37:17.122Z Wa(03)+ host-13732 
2023-10-28T04:37:17.122Z Wa(03)+ host-13732 Stderr = 
2023-10-28T04:37:17.122Z Wa(03)+ host-13732 
2023-10-28T04:37:17.125Z Wa(03) host-13732 <vpxd> Service pre-start command's stderr: Traceback (most recent call last):
2023-10-28T04:37:17.125Z Wa(03)+ host-13732   File "/usr/lib/vmware-vpx/py/vpxd-prestart.py", line 68, in <module>
2023-10-28T04:37:17.125Z Wa(03)+ host-13732     patch_vpxd_prop()
2023-10-28T04:37:17.125Z Wa(03)+ host-13732   File "/usr/lib/vmware-vpx/py/vpxd_update.py", line 314, in patch_vpxd_prop
2023-10-28T04:37:17.125Z Wa(03)+ host-13732     updateGoscSpecDecertInB2BOrNDU()
2023-10-28T04:37:17.125Z Wa(03)+ host-13732   File "/usr/lib/vmware-vpx/py/vpxd_update.py", line 271, in updateGoscSpecDecertInB2BOrNDU
2023-10-28T04:37:17.125Z Wa(03)+ host-13732     decert.create()
2023-10-28T04:37:17.125Z Wa(03)+ host-13732   File "/usr/lib/vmware-vpx/py/data_encipherment_cert_utils.py", line 183, in create
2023-10-28T04:37:17.125Z Wa(03)+ host-13732     self._gen_cert()
2023-10-28T04:37:17.125Z Wa(03)+ host-13732   File "/usr/lib/vmware-vpx/py/data_encipherment_cert_utils.py", line 147, in _gen_cert
2023-10-28T04:37:17.125Z Wa(03)+ host-13732     raise e
2023-10-28T04:37:17.125Z Wa(03)+ host-13732   File "/usr/lib/vmware-vpx/py/data_encipherment_cert_utils.py", line 135, in _gen_cert
2023-10-28T04:37:17.125Z Wa(03)+ host-13732     invoke_command(cmd)
2023-10-28T04:37:17.125Z Wa(03)+ host-13732   File "/usr/lib/vmware/site-packages/cis/utils.py", line 369, in invoke_command
2023-10-28T04:37:17.125Z Wa(03)+ host-13732     (cmd, stderr))
2023-10-28T04:37:17.125Z Wa(03)+ host-13732 cis.exceptions.InvokeCommandException: {
2023-10-28T04:37:17.125Z Wa(03)+ host-13732     "detail": [
2023-10-28T04:37:17.125Z Wa(03)+ host-13732         {
2023-10-28T04:37:17.125Z Wa(03)+ host-13732             "id": "install.ciscommon.command.errinvoke",
2023-10-28T04:37:17.125Z Wa(03)+ host-13732             "translatable": "An error occurred while invoking external command : '%(0)s'",
2023-10-28T04:37:17.125Z Wa(03)+ host-13732             "args": [
2023-10-28T04:37:17.125Z Wa(03)+ host-13732                 "Command: ['/usr/lib/vmware-vmca/bin/certool', '--server=FQDNofvCenter', '--genCIScert', '--dataencipherment', '--privkey=/etc/vmware-vpx/ssl/tmp-data-encipherment.key', '--cert=/etc/vmware-vpx/ssl/tmp-data-encipherment.crt', '--Name=data-encipherment', '--IPAddress=FQDNofvCenter']\nStderr: "
2023-10-28T04:37:17.125Z Wa(03)+ host-13732             ],
2023-10-28T04:37:17.125Z Wa(03)+ host-13732             "localized": "An error occurred while invoking external command : 'Command: ['/usr/lib/vmware-vmca/bin/certool', '--server=FQDNofvCenter', '--genCIScert', '--dataencipherment', '--privkey=/etc/vmware-vpx/ssl/tmp-data-encipherment.key', '--cert=/etc/vmware-vpx/ssl/tmp-data-encipherment.crt', '--Name=data-encipherment', '--IPAddress=FQDNofvCenter']\nStderr: '"
2023-10-28T04:37:17.125Z Wa(03)+ host-13732         },
2023-10-28T04:37:17.125Z Wa(03)+ host-13732         {
2023-10-28T04:37:17.125Z Wa(03)+ host-13732             "id": "upgrade.vpxd.cert.create.failed",
2023-10-28T04:37:17.125Z Wa(03)+ host-13732             "translatable": "Failed to create data encipherment cert with hostname/ip %(0)s.",
2023-10-28T04:37:17.125Z Wa(03)+ host-13732             "args": [
2023-10-28T04:37:17.125Z Wa(03)+ host-13732                 "vcenter.example.com"
2023-10-28T04:37:17.125Z Wa(03)+ host-13732             ],
2023-10-28T04:37:17.125Z Wa(03)+ host-13732             "localized": "Failed to create data encipherment cert with hostname/ip vcenter.example.com."
2023-10-28T04:37:17.125Z Wa(03)+ host-13732         }
2023-10-28T04:37:17.125Z Wa(03)+ host-13732     ],
2023-10-28T04:37:17.125Z Wa(03)+ host-13732     "componentKey": null,
2023-10-28T04:37:17.125Z Wa(03)+ host-13732     "problemId": null,
2023-10-28T04:37:17.125Z Wa(03)+ host-13732     "resolution": null
2023-10-28T04:37:17.125Z Wa(03)+ host-13732 }
2023-10-28T04:37:17.125Z Wa(03)+ host-13732 
2023-10-28T04:37:17.189Z Er(02) host-13732 <vpxd> Service pre-start command failed with exit code 1.
2023-10-28T04:37:17.189Z Wa(03) host-13732 [ReadSvcSubStartupData] No startup information from vpxd.

Environment

VMware vCenter Server 7.0
VMware vCenter Server 8.0

Cause

Patching will fail if the "data-encipherment" certificate is already expired or is about to expire within 1 year as the patch script contains code to replace the data-encipherment certificate if it is about to expire.

Resolution

1. Revert to the previous snapshot.
2. Renew the data-encryption certificate using the methods explained in the following KB article:
   How to replace an expired data-encipherment certificate on vCenter Server