Scheduled file-based backups are failing in VCSA 7.x/8.x while manual backups are working fine

Products

• VMware vCenter Server 7.0
• VMware vCenter Server 8.0

Issue/Introduction

• Scheduled file-based backups do not run in vCenter Server Appliance.
• However, manual file-based backups are working fine.
• You may also have issues in logging into the VAMI client using SSO account  while login with root is ok.
• OVF operations (export, deploy) are also failing with error "Unable to authenticate user"
• When reviewing the backup scheduler log /var/log/vmware/applmgmt/backupScheduler.log you find errors similar to the following excerpt:

  YYYY-MM-DDThh:mm:ss:xxxx [0] [MainProcess:PID-413] [Scheduler::ExecScheduleRun:Scheduler.py:138] ERROR: Failed to issue the Schedules.run request. Exception: {challenge : None, messages : [LocalizableMessage(id='vapi.security.authentication.invalid', default_message='Unable to authenticate user', args=[], params=None, localized=None)], data : None, error_type : UNAUTHENTICATED}
  Traceback (most recent call last):
    File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/Scheduler.py", line 133, in ExecScheduleRun
      status = svc_handle.run(scheduleId, comment='SCHEDULED')
    File "/usr/lib/applmgmt/pyclient/applmgmt_client-1.0-py2.7.egg/com/vmware/appliance/recovery/backup_client.py", line 1171, in run
      'comment': comment,
    File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/stub.py", line 345, in _invoke
      return self._api_interface.native_invoke(ctx, _method_name, kwargs)
    File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/stub.py", line 298, in native_invoke
      self._rest_converter_mode)
  com.vmware.vapi.std.errors_client.Unauthenticated: {challenge : None, messages : [LocalizableMessage(id='vapi.security.authentication.invalid', default_message='Unable to authenticate user', args=[], params=None, localized=None)], data : None, error_type : UNAUTHENTICATED}

   
• The log of the appliance management service, /var/log/vmware/applmgmt/applmgmt.log, contains errors like the ones below:

  YYYY-MM-DDTHH:MM:SS AM UTC [3179]INFO:vmware.appliance.backup_restore.schedule_impl:Schedule with id 'default' updated successfully.
  YYYY-MM-DDTHH:MM:SS AM UTC [3179]ERROR:vmware.appliance.extensions.authentication.authentication_sso:Unhandled exception during SAML token validation
  Traceback (most recent call last):
    File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authentication/authentication_sso.py", line 507, in validate
      self.validate_certificate()
    File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authentication/authentication_sso.py", line 674, in validate_certificate
      self.add_x509_pem_header(c)) for c in certsFromToken]
    File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authentication/authentication_sso.py", line 674, in <listcomp>
      self.add_x509_pem_header(c)) for c in certsFromToken]
    File "/usr/lib/python3.7/site-packages/OpenSSL/crypto.py", line 1825, in load_certificate
      _raise_current_error()
    File "/usr/lib/python3.7/site-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
      raise exception_type(errors)
  OpenSSL.crypto.Error: []
  YYYY-MM-DDTHH:MM:SS AM UTC [3179]ERROR:vmware.appliance.vapi.auth:Could not parse HOK Token
  Traceback (most recent call last):
    File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authentication/authentication_sso.py", line 507, in validate
      self.validate_certificate()
    File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authentication/authentication_sso.py", line 674, in validate_certificate
      self.add_x509_pem_header(c)) for c in certsFromToken]
    File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authentication/authentication_sso.py", line 674, in <listcomp>
      self.add_x509_pem_header(c)) for c in certsFromToken]
    File "/usr/lib/python3.7/site-packages/OpenSSL/crypto.py", line 1825, in load_certificate
      _raise_current_error()
    File "/usr/lib/python3.7/site-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
      raise exception_type(errors)
  OpenSSL.crypto.Error: []

 

Environment

• vSphere vCenter 7.0.x
• vSphere vCenter 8.0.x

Cause

Despite the appearance this is not a certificate issue.

The authentication is not getting completed while the scheduler is trying to trigger the backup due to an unhandled crypto error in from the OpenSSL crypto library.

 

Resolution

This issue is fixed in vCenter Server 8.0 Update 3 and later versions. In case you cannot upgrade to this version, you can apply the workaround below to fix the problem.

1. Open an SSH connection with the vCenter Server Appliance and login with the user root.
2. Create a backup copy  of the script /usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authentication/authentication_sso.py

   # cp /usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authentication/authentication_sso.py /usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authentication/authentication_sso.py.original

3. Replace the authentication_sso.py with the one attached to this KB
4. Restart the applmgmt service:

   # service-control --restart applmgmt

5. Open the VAMI UI https://<vCenter-FQDN>:5480 in your web browser and login using SSO credentials to verify the UI is working fine.
6. Schedule a file-based backup to confirm the issue is resolved.