Scheduled backups are failing for VCSA via VAMI is failing with error "Requested SSO authentication but SSO authentication module is not available"
search cancel

Scheduled backups are failing for VCSA via VAMI is failing with error "Requested SSO authentication but SSO authentication module is not available"

book

Article ID: 387630

calendar_today

Updated On:

Products

VMware vCenter Server VMware vCenter Server 7.0 VMware vCenter Server 8.0

Issue/Introduction

  • Scheduled backups on vCenter Server Appliance (VCSA) via vCenter Appliance Management Interface (VAMI) is failing
  • Issue is not seen while performing manual backup via VAMI
  • /var/log/vmware/applmgmt/backupScheduler.log

YYYY-MM-DDTHH:MM:SS  [0] [MainProcess:PID-2458722] [serviceAccountUtil::readPassword:serviceAccountUtil.py:82] INFO: Fetching the service account password from file
YYYY-MM-DDTHH:MM:SS  [0] [MainProcess:PID-2458722] [VapiClientHelper::get_vapi_stub_with_saml_auth:VapiClientHelper.py:68] INFO: Getting ephemeral certificate
YYYY-MM-DDTHH:MM:SS  [0] [MainProcess:PID-2458722] [VapiClientHelper::get_saml_token_with_svc_user:VapiClientHelper.py:109] INFO: Getting saml token with svc user
YYYY-MM-DDTHH:MM:SS [0] [MainProcess:PID-2458722] [VapiClientHelper::get_vapi_stub_with_saml_auth:VapiClientHelper.py:88] INFO: Obtained saml token with svc user. Getting the vapi stub.
YYYY-MM-DDTHH:MM:SS [0] [MainProcess:PID-2458722] [Scheduler::ExecScheduleRun:Scheduler.py:137] ERROR: Failed to issue the Schedules.run request. Exception: {challenge : None, messages : [LocalizableMessage(id='vapi.security.authentication.invalid', default_message='Unable to authenticate user', args=[], params=None, localized=None)], data : None, error_type : UNAUTHENTICATED}
Traceback (most recent call last):
  File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/Scheduler.py", line 133, in ExecScheduleRun
    status = svc_handle.run(scheduleId, comment='SCHEDULED')
  File "/usr/lib/applmgmt/pyclient/applmgmt_client-1.0-py2.7.egg/com/vmware/appliance/recovery/backup_client.py", line 1186, in run
    return self._invoke('run',
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/stub.py", line 345, in _invoke
    return self._api_interface.native_invoke(ctx, _method_name, kwargs)
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/stub.py", line 295, in native_invoke
    raise TypeConverter.convert_to_python(method_result.error,  # pylint: disable=E0702
com.vmware.vapi.std.errors_client.Unauthenticated: {challenge : None, messages : [LocalizableMessage(id='vapi.security.authentication.invalid', default_message='Unable to authenticate user', args=[], params=None, localized=None)], data : None, error_type : UNAUTHENTICATED}

  • /var/log/vmware/applmgmt/applmgmt.log

YYYY-MM-DDTHH:MM:SS [2441130]INFO:vmware.vherd.transport.vapi:Provider config only patterns ['com.vmware.vcenter.deployment.import_history', 'com.vmware.appliance.update', 'com.vmware.appliance.system.version', 'com.vmware.cis.session', 'com.vmware.appliance.health', 'com.vmware.appliance.local_accounts', 'com.vmware.vapi', 'com.vmware.vcenter.deployment']
YYYY-MM-DDTHH:MM:SS [2441130]INFO:vmware.vherd.transport.vapi:Provider config block patterns []
YYYY-MM-DDTHH:MM:SS [2441130]DEBUG:vmware.vherd.base.authorization_local:Local authorization initialized
YYYY-MM-DDTHH:MM:SS [2441130]WARNING:vmware.appliance.vapi.auth:SSO Authentication library not available, make sure applmgmt-cloudvm-*.rpm is installed

YYYY-MM-DDTHH:MM:SS [2441130]INFO:vmware.appliance.backup_restore.ScheduleManager:Schedule created successfully.
YYYY-MM-DDTHH:MM:SS [2441130]INFO:vmware.appliance.vapi.auth:Authorization request for service_id: com.vmware.appliance.recovery.backup.job.details, operation_id: list
YYYY-MM-DDTHH:MM:SS [2441130]DEBUG:vmware.vherd.base.authorization_local:Verify privileges user (root) privilege ['ModifyConfiguration']
YYYY-MM-DDTHH:MM:SS [2441130]DEBUG:root:Validated user privileges in localstore or SSO

YYYY-MM-DDTHH:MM:SS [2441130]DEBUG:vmware.appliance.update.update_state:In State._get using state file /etc/applmgmt/appliance/software_update_state.conf
YYYY-MM-DDTHH:MM:SS [2441130]DEBUG:vmware.appliance.update.update_state:In State._get using state file /etc/applmgmt/appliance/software_update_state.conf
YYYY-MM-DDTHH:MM:SS [2441130]ERROR:vmware.appliance.vapi.auth:Requested SSO authentication but SSO authentication module is not available.

Cause

This issue is caused due to invalid or stale STS certificate on the VCSA

Resolution

Validate the stale/incorrect certificate on VCSA

  • Log in to VCSA via ssh session
  • Execute the below command

# /opt/likewise/bin/ldapsearch -b "cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local" -s sub -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -W | grep -E 'cn=TenantCredential|cn=TrustedCertificateChains'

Note: In this case, the SSO domain is vsphere.local and the dc values needs to be modified based on the domain

Sample output (expected state):

# /opt/likewise/bin/ldapsearch -b "cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local" -s sub -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -W | grep -E 'cn=TenantCredential|cn=TrustedCertificateChains'

dn: cn=TenantCredential-1,cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local
--
dn: cn=TrustedCertificateChains,cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local
--
dn: cn=TrustedCertChain-1,cn=TrustedCertificateChains,cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local

In order to resolve the issue, proceed with the replacement of STS certificate

  • Connect to VCSA via ssh session with the root user
  • Download the attached fixsts.sh script from this article and upload to the impacted VCSA to the /tmp folder

    • If the connection to upload to the vCenter by the SCP client is rejected, run this from an SSH session to the vCenter:

# chsh -s /bin/bash root

  • Navigate to the /tmp directory:

cd /tmp

  • Make the file executable

chmod +x fixsts.sh

  • Run the script

./fixsts.sh

  • Restart services on VCSA (in case of Enhanced Linked Mode, service restart is required on all VCSA's) in your SSO domain by using below commands:

service-control --stop --all && service-control --start --all

Additional Information

Attachments

fixsts.sh get_app
Powered by Wolken Software