Scheduled backup of vCenter Server is not triggered after replacement of root and sts certificates.
search cancel

Scheduled backup of vCenter Server is not triggered after replacement of root and sts certificates.

book

Article ID: 398319

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

- Scheduled backups are not triggered.
  However, taking a manual file-based backup works fine.

- /var/log/vmware/applmgmt/backupScheduler.log shows

YYYY-mm-ddTHH:MM:SS.xxx [0] [MainProcess:PID-23451] [Scheduler::ExecScheduleRun:Scheduler.py:138] ERROR: Failed to issue the Schedules.run request. Exception: {challenge : None, messages : [LocalizableMessage(id='vapi.security.authentication.invalid', default_message='Unable to authenticate user', args=[], params=None, localized=None)], data : None, error_type : UNAUTHENTICATED}
Traceback (most recent call last):
  File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/Scheduler.py", line 133, in ExecScheduleRun
    status = svc_handle.run(scheduleId, comment='SCHEDULED')
  File "/usr/lib/applmgmt/pyclient/applmgmt_client-1.0-py2.7.egg/com/vmware/appliance/recovery/backup_client.py", line 1189, in run
    'comment': comment,
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/stub.py", line 345, in _invoke
    return self._api_interface.native_invoke(ctx, _method_name, kwargs)
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/stub.py", line 298, in native_invoke
    self._rest_converter_mode)
com.vmware.vapi.std.errors_client.Unauthenticated: {challenge : None, messages : [LocalizableMessage(id='vapi.security.authentication.invalid', default_message='Unable to authenticate user', args=[], params=None, localized=None)], data : None, error_type : UNAUTHENTICATED}

 

- /var/log/vmware/applmgmt/applmgmt.log shows

YYYY-mm-ddTHH:MM:SS AM UTC [10376]DEBUG:vmware.appliance.extensions.authentication.authentication_sso:Downloading trusted certs from url : http://localhost:7080/idm/tenant/vsphere.local/certificates?scope=TENANT
YYYY-mm-ddTHH:MM:SS AM UTC [10376]DEBUG:vmware.appliance.extensions.authentication.authentication_sso:Downloading trusted certs from url : http://localhost:7080/idm/tenant/vsphere.local/certificates?scope=TENANT
YYYY-mm-ddTHH:MM:SS AM UTC [10376]ERROR:vmware.appliance.vapi.auth:Could not parse HOK Token
Traceback (most recent call last):
  File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authentication/authentication_sso.py", line 507, in validate
    self.validate_certificate()
  File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authentication/authentication_sso.py", line 710, in validate_certificate
    'One or more certificates cannot be verified.')
vmware.appliance.extensions.authentication.authentication_sso.AuthenticationError: One or more certificates cannot be verified.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/applmgmt/vapi/py/vmware/appliance/vapi/auth.py", line 251, in authenticate
    username = token.username
  File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authentication/authentication_sso.py", line 487, in username
    return self.get_name_id().value
  File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authentication/authentication_sso.py", line 983, in get_name_id
    '//saml2:Subject/saml2:NameID', self.reference)
  File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authentication/authentication_sso.py", line 478, in reference
    self.validate()
  File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authentication/authentication_sso.py", line 1213, in validate
    reference = super(HolderOfKeyToken, self).validate()
  File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authentication/authentication_sso.py", line 509, in validate
    self.validate_certificate()
  File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authentication/authentication_sso.py", line 710, in validate_certificate
    'One or more certificates cannot be verified.')
vmware.appliance.extensions.authentication.authentication_sso.AuthenticationError: One or more certificates cannot be verified.

- Retrieving STS certificates shows multiple root certificates with same subject name

curl -s http://localhost:7080/idm/tenant/$(/usr/lib/vmware-vmafd/bin/vmafd-cli get-domain-name --server-name localhost)/certificates?scope=TENANT | jq  .[].certificates[].encoded  | while read -r line; do echo -e $line | tr -d '"' | openssl x509 -noout -text | egrep --no-group-separator -A1 "Subject:|Issuer:|Key Identifier" ;done

Environment

  •  vCenter Server 7.0.x
  •  vCenter Server 8.0.x

Cause

If there are multiple root CA certificates with the same subject name, certificate validation for authentication fails when performing a scheduled backup.

Resolution

Run vCert script to renew STS certificate and remove old entries.

For using vCert script, see vCert - expired certificate replacement script

./vCert.py --run config/manage_cert/sts_signing/op_vmca-signed-cert.yaml

 

Powered by Wolken Software