Patching vCenter Server to 8.0 U3 fails with error "Pre-Install failed for vmidentity:Expand"

Products

• VMware vCenter Server 8.0

Issue/Introduction

• Patching vCenter Server to 8.0 U3 fails with error "Pre-install failed for vmidentity:Expand".
  
  

 

• Log file /var/log/vmware/applmgmt/Patchrunner.log will show below entries:
  
  /var/log/vmware/applmgmt/Patchrunner.log:
  

  vmidentity:Expand INFO vmidentity.utils Executing: ['/usr/lib/vmware-vmafd/bin/vecs-cli', 'entry', 'list', '--store', 'TRUSTED_ROOTS']
vmidentity:Expand INFO vmidentity.utils get all certs completed.
vmidentity:Expand INFO vmidentity Found <> certs in TRUSTED_ROOTS store
vmidentity:Expand INFO vmidentity CN of cert_3=ssoserver
vmidentity:Expand INFO vmidentity Found ssoserver cert in TRUSTED_ROOTS, This will be deleted from store
vmidentity:Expand INFO vmidentity.utils Deleting cert from TRUSTED_ROOTS VECS store
vmidentity:Expand ERROR vmidentity.utils Failed to execute command '['/usr/lib/vmware-vmafd/bin/dir-cli', 'trustedcert', 'unpublish', '--cert', '/storage/seat/software-updateub8jty50/stage/scripts/patches/payload/components-script/vmidentity/<Cert_filename.pem>', '--login', '<VC FQDN>']'
vmidentity:Expand ERROR vmidentity.utils dir-cli failed. Error 1168: Operation failed with error ERROR_NOT_FOUND (1168)

  vmidentity:Expand ERROR vmidentity Something went wrong while reading certs from TRUSTED_ROOTS or deleting ssoserver cert: %s
vmidentity:Expand ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'vmidentity:Expand' failed.
Traceback (most recent call last):
  File "/storage/seat/software-updateub8jty50/stage/scripts/patches/payload/components-script/vmidentity/utils.py", line 275, in delete_ssoserver_cert
    raise Exception("Failed to execute dir-cli unpublish cert")
Exception: Failed to execute dir-cli unpublish cert

Environment

vCenter Server 8.0 U3

Resolution

This issue is resolved in vCenter Server 8.0 U3a.

 

Workaround

To workaround this issue, remove the Non-CA Certificate with CN "ssoserver" from TRUSTED_ROOTS of vCenter Server by following any of the below methods:

• Automated Method - Clean-up using a shell script
• Manual Method - Clean-up using vecs-cli command
  
  

Scripted Workaround

To workaround this issue using the included fix_ssoserver_cert.sh script, follow the steps below:

• Download the attached fix_ssoserver_cert.sh and copy it to the vCenter Appliance
  
  
• To list certificates in the TRUSTED_ROOTS store, run the script with the "list" flag

bash ./fix_ssoserver_cert.sh list

• To clean up any certificates with ssoserver anywhere in the subject, run the script with the "clean" flag.

bash ./fix_ssoserver_cert.sh clean

• Verify the ssoserver certificate is removed by re-running the script with the "list" flag
  
  
• Resume the vCenter Server update.

Manual Workaround

To workaround this issue using vecs-cli to remove the Non-CA Certificate with CN "ssoserver" from TRUSTED_ROOTS, use the below steps :

• SSH to the vCenter Server
  
  
• List the Certificates and identify the Alias of Non-CA Certificate with CN=ssoserver
  
  /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | egrep 'Alias|ssoserver|Key Usage' -A 1 | egrep -v 'Entry type|--'
  
  
• Take a backup of the certificate
  
  /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store TRUSTED_ROOTS --alias <Alias> --output /var/tmp/non_ca_ssoserver.crt

Note: Replace <Alias> with the Alias ID identified from above step.
  
  
• Remove the certificate from VECS Store
  
  /usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store TRUSTED_ROOTS --alias <Alias> -y
  
  
• Perform Certificate list again and make sure the certificate is removed
  
  /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | egrep 'Alias|ssoserver|Key Usage' -A 1 | egrep -v 'Entry type|--'
  
  
• Resume the vCenter Server update