Failed to force refresh TRUSTED_ROOTS, Error : 11" while upgrading VCSA
search cancel

Failed to force refresh TRUSTED_ROOTS, Error : 11" while upgrading VCSA

book

Article ID: 325029

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

This KB article is to fix the corrupted TRUSTED_ROOTS or TRUSTED_ROOT_CRLS store

vCenter Server Appliance upgrade from 7.0 to 8.0 fails with the below error messages and log snippets.

  • You will see the similar log entries as below

/var/log/firstboot/vmafd-firstboot.py_<pid>_stderr.log

YYYY-MM-DDTHH:MM:SS.198Z  Exception: Traceback (most recent call last):
  File "/usr/lib/vmware-vmafd/firstboot/vmafd-firstboot.py", line 179, in main
    controller.client_mode()
  File "/usr/lib/vmware-vmafd/firstboot/vmafd-firstboot.py", line 79, in client_mode
    service.client_mode()
  File "/usr/lib/vmware-vmafd/firstboot/identityinstall/vmafdUpgrade.py", line 269, in client_mode
    self.post_init()
  File "/usr/lib/vmware-vmafd/firstboot/identityinstall/vmafdUpgrade.py", line 255, in post_init
    self.vecs_force_refresh()
  File "/usr/lib/vmware-vmafd/firstboot/identityinstall/vmafdInstall.py", line 681, in vecs_force_refresh
    problemId = "install.vmafd.vecs_force_refresh_failed")
cis.baseCISException.BaseInstallException: {
    "componentKey": "vmafd",
    "detail": [
        {
            "args": [
                11
            ],
            "translatable": "Failed to force refresh TRUSTED_ROOTS, Error : %(0)d",
            "localized": "Failed to force refresh TRUSTED_ROOTS, Error : 11",
            "id": "install.vmafd.vecs_force_refresh_failed"
        }
    ],
    "resolution": {
        "translatable": "Please search of these symptoms in the VMware Knowledge Base for any known issues and possible workarounds. If none can be found, please collect a support bundle and open a support request.",
        "localized": "Please search of these symptoms in the VMware Knowledge Base for any known issues and possible workarounds. If none can be found, please collect a support bundle and open a support request.",
        "id": "install.vmafd.vecs_force_refresh_failed.resolution"
    },
    "problemId": "install.vmafd.vecs_force_refresh_failed"

    
/var/log/vmware/vmafdd/vmafdd-syslog.log


YYYY-MM-DDTHH:MM.183996+00:00 notice vmafdd  t@140047053448960: VmAfdProcessCACerts: force flushing.
YYYY-MM-DDTHH:MM.101369+00:00 notice vmafdd  t@140047223973632: Failed to update trusted roots. Error [11]
YYYY-MM-DDTHH:MM.101200+00:00 err vmafdd  t@140047223973632: [Error - 11, ../../../server/vmafd/rootfetch.c:270]
YYYY-MM-DDTHH:MM.101008+00:00 err vmafdd  t@140047223973632: [Error - 11, ../../../server/vmafd/rootfetch.c:762]
YYYY-MM-DDTHH:MM:23.100790+00:00 err vmafdd  t@140047223973632: [Error - 11, ../../../server/vmafd/vecsserviceapi.c:426]
YYYY-MM-DDTHH:MM.071140+00:00 notice vmafdd  t@140047932479232: vmafdd: started!
YYYY-MM-DDTHH:MM.070978+00:00 notice vmafdd  t@140047932479232: Started CDC Cache Thread successfully, CdcInitCdcCacheUpdate
YYYY-MM-DDTHH:MM.070814+00:00 notice vmafdd  t@140047932479232: Starting CDC Caching Thread, CdcInitCdcCacheUpdate
YYYY-MM-DDTHH:MM.070651+00:00 notice vmafdd  t@140047932479232: Started CDC State Machine Thread successfully, CdcInitStateMachine
YYYY-MM-DDTHH:MM.070486+00:00 notice vmafdd  t@140047932479232: Starting the CDC State machine, CdcInitStateMachine
 
  • Listing VECS Store on source vCenter Server fails with error "ERROR_BAD_FORMAT" 
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOT_CRLS
Number of entries in store : 6
vecs-cli failed. Error 11: Possible errors:
LDAP error: Administrative limit exceeded
Win Error: Operation failed with error ERROR_BAD_FORMAT (11)
 



Environment

VMware vCenter Server Appliance 7.0.x
VMware vCenter Server Appliance 8.0.x

Cause

This is caused due to corrupted VECS certificate store


Resolution

  • To verify the available certificate stores on the source vCenter Server, run the following command: "/usr/lib/vmware-vmafd/bin/vecs-cli store list".

Below is the expected output: vCenter Server 7.0 & 8.0

MACHINE_SSL_CERT
TRUSTED_ROOTS
TRUSTED_ROOT_CRLS
machine
vsphere-webclient
vpxd
vpxd-extension
hvc
data-encipherment
SMS
APPLMGMT_PASSWORD
wcp
BACKUP_STORE

  • Run the following commands to backup certificates and keys from each store: (Note: This step is very important)

TRUSTED_ROOTS and BACKUP_STORE:
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS > /var/tmp/root-cert.txt
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store BACKUP_STORE > /var/tmp/BACKUP_STORE_data.txt

MACHINE_SSL_CERT:
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output /var/tmp/Machine_SSL.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output /var/tmp/Machine_SSL.key

vpxd-extension:
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /var/tmp/vpxd-extension.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vpxd-extension --alias vpxd-extension --output /var/tmp/vpxd-extension.key

vpxd:
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd --alias vpxd --output /var/tmp/vpxd.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vpxd --alias vpxd --output /var/tmp/vpxd.key

machine:
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store machine --alias machine --output /var/tmp/machine.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store machine --alias machine --output /var/tmp/machine.key

vsphere-webclient:
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vsphere-webclient --alias vsphere-webclient --output /var/tmp/vsphere-webclient.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vsphere-webclient --alias vsphere-webclient --output /var/tmp/vsphere-webclient.key

SMS:
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store SMS --alias sms_self_signed --output /var/tmp/sms_self_signed_new.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store SMS --alias sms_self_signed --output /var/tmp/sms_self_signed_new.key

hvc:
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store hvc --alias hvc --output /var/tmp/hvc.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store hvc --alias hvc --output /var/tmp/hvc.key

data-encipherment:
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store data-encipherment --alias data-encipherment --output /var/tmp/data-encipherment.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store data-encipherment --alias data-encipherment --output /var/tmp/data-encipherment.key

APPLMGMT_PASSWORD:
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store APPLMGMT_PASSWORD > /var/tmp/applmgmt.txt

wcp:
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store wcp --alias wcp --output /var/tmp/wcp.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store wcp --alias wcp --output /var/tmp/wcp.key

  • List permission of each store and note the user and access granted to each user.

    /usr/lib/vmware-vmafd/bin/vecs-cli store get-permissions --name <store name>

    Example: /usr/lib/vmware-vmafd/bin/vecs-cli store get-permissions --name machine_ssl_Cert

  • Stop VMAFDD Service after ensuring the backup files are available.

    service-control --stop vmafdd

  • Rename VECS Database

mv /storage/db/vmware-vmafd/afd.db /storage/db/vmware-vmafd/afd.db_old

  • Start VMAFDD Service which will create a new VECS Database in /storage/db/vmware-vmafd/

service-control --start vmafdd

  • List the stores on new DB which will show only default stores :

    Execute: /usr/lib/vmware-vmafd/bin/vecs-cli store list

    MACHINE_SSL_CERT
    TRUSTED_ROOTS
    TRUSTED_ROOT_CRLS

The new STORE will only have MACHINE_SSL_CERT, TRUSTED_ROOTS, TRUSTED_ROOT_CRLS however the store information will not hold any permissions. We have to manually create the other certificate stores machine, vpxd, vpxd-extension,vsphere-webclient and so on.

  • Follow below steps to create and provide the permissions on the certificate store

Add permissions for MACHINE_SSL_CERT store :

/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name MACHINE_SSL_CERT --user vlcm --grant read
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name MACHINE_SSL_CERT --user updatemgr --grant read
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name MACHINE_SSL_CERT --user vsphere-ui --grant read
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name MACHINE_SSL_CERT --user vpxd --grant read
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name MACHINE_SSL_CERT --user vpostgres --grant read
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name MACHINE_SSL_CERT --user vsm --grant read


Add permissions for TRUSTED_ROOTS and TRUSTED_ROOT_CRLS store :
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name TRUSTED_ROOTS --user vpxd --grant read
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name TRUSTED_ROOTS --user sps --grant write
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name TRUSTED_ROOT_CRLS --user vpxd --grant read
 

  • Create solution user certificate stores and grant permissions as below :

machine:
/usr/lib/vmware-vmafd/bin/vecs-cli store create --name machine
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name machine --user infraprofile --grant write
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name machine --user vpxd --grant read
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name machine --user vsan-health --grant read
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name machine --user certauth --grant write
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name machine --user certmgr --grant write
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name machine --user observability --grant read

vpxd:
/usr/lib/vmware-vmafd/bin/vecs-cli store create --name vpxd
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name vpxd --user vpxd --grant read

vpxd-extension:
/usr/lib/vmware-vmafd/bin/vecs-cli store create --name vpxd-extension
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name vpxd-extension --user vlcm --grant read
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name vpxd-extension --user wcp --grant read
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name vpxd-extension --user deploy --grant read
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name vpxd-extension --user infraprofile --grant write
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name vpxd-extension --user updatemgr --grant read
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name vpxd-extension --user vsphere-ui --grant read
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name vpxd-extension --user vpxd --grant read
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name vpxd-extension --user analytics --grant read
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name vpxd-extension --user vsm --grant read
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name vpxd-extension --user vsan-health --grant read
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name vpxd-extension --user imagebuilder --grant read
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name vpxd-extension --user content-library --grant read
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name vpxd-extension --user eam --grant read
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name vpxd-extension --user sps --grant write
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name vpxd-extension --user vstatsuser --grant read

vsphere-webclient:
/usr/lib/vmware-vmafd/bin/vecs-cli store create --name vsphere-webclient
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name vsphere-webclient --user infraprofile --grant write
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name vsphere-webclient --user vsphere-ui --grant read
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name vsphere-webclient --user vpxd --grant read
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name vsphere-webclient --user perfcharts --grant read
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name vsphere-webclient --user vapiEndpoint --grant read
 /usr/lib/vmware-vmafd/bin/vecs-cli store permission --name vsphere-webclient --user analytics --grant read

SMS:
/usr/lib/vmware-vmafd/bin/vecs-cli store create --name SMS
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name SMS --user vpxd --grant read
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name SMS --user deploy --grant read

hvc:
/usr/lib/vmware-vmafd/bin/vecs-cli store create --name hvc
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name hvc --user vpxd --grant write
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name hvc --user hvc --grant write

wcp:
/usr/lib/vmware-vmafd/bin/vecs-cli store create --name wcp
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name wcp --user content-library --grant read
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name wcp --user wcp --grant read
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name wcp --user vpxd --grant read

data-encipherment:
/usr/lib/vmware-vmafd/bin/vecs-cli store create --name data-encipherment
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name data-encipherment --user vpxd --grant read

APPLMGMT_PASSWORD:
/usr/lib/vmware-vmafd/bin/vecs-cli store create --name APPLMGMT_PASSWORD
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name APPLMGMT_PASSWORD --user vpxd --grant read

  • Restore the certificates to the respective certificate stores:

/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store MACHINE_SSL_CERT --alias __MACHINE_CERT --cert /var/tmp/Machine_SSL.crt --key /var/tmp/Machine_SSL.key
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store machine --alias machine --cert /var/tmp/machine.crt --key /var/tmp/machine.key
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store vpxd --alias vpxd --cert /var/tmp/vpxd.crt --key /var/tmp/vpxd.key
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store vpxd-extension --alias vpxd-extension --cert /var/tmp/vpxd-extension.crt --key /var/tmp/vpxd-extension.key
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store vsphere-webclient --alias vsphere-webclient --cert /var/tmp/vsphere-webclient.crt --key /var/tmp/vsphere-webclient.key
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store SMS --alias sms_self_signed --cert /var/tmp/sms_self_signed_new.crt --key /var/tmp/sms_self_signed_new.key
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store hvc --alias hvc --cert /var/tmp/hvc_new.crt --key /var/tmp/hvc_new.key
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store wcp --alias wcp --cert /var/tmp/wcp_new.crt --key /var/tmp/wcp_new.key
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store data-encipherment --alias data-encipherment  --cert  /var/tmp/data-encipherment.crt --key /var/tmp/data-encipherment.key

  • List all certificate stores using below commands and make sure certificates are listed

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | less
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT --text | less
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store machine --text | less
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vpxd --text | less
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vpxd-extension --text | less
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vsphere-webclient --text | less
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store SMS --text | less
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOT_CRLS --text | less
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store wcp --text | less
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store hvc --text | less
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store data-encipherment --text | less

  • Restart all services

    service-control --stop --all
    service-control --start --all

**************************************************************************************************************************

Note: Recreating BACKUP_STORE is not mandatory, if necessary use below commands to create the store
/usr/lib/vmware-vmafd/bin/vecs-cli store create --name BACKUP_STORE
/usr/lib/vmware-vmafd/bin/vecs-cli store permission --name BACKUP_STORE --user vpxd --grant read

Add each certificate from the backup /var/tmp/BACKUP_STORE_data.txt with respective alias listed in backup file using below command:

/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store BACKUP_STORE --alias <respective alias from backup file> --cert <certificate created from backup of BACKUP_STORE>

Certificates in TRUSTED ROOTS, TRUSTED_ROOT_CRLS will be auto populated once all the stores are created.

*************************************************************************************************************************************

  • Retry the vCenter Server upgrade

Additional Information

Impact/Risks:
Ensure to take backup of all the certificates in the store before proceeding along with Snapshot/Backup of vCenter Server.

Powered by Wolken Software