"Unable to authenticate user (Server error id: 'vapi.security.authentication.invalid')", Trying to configure automated backup of vCenter server via API calls (using script) fails

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• Unable to run a automated script to backup the VCSA
• PowerCLI script shows below error message: 

Error:  A server error occurred: 'com.vmware.vapi.std.errors.unauthenticated': Unable to authenticate user (Server error id:
'vapi.security.authentication.invalid'). Check $Error[0].Exception.ServerError for more details.
ERROR:vmware.appliance.vapi.auth:Requested SSO authentication but SSO authentication module is not available

• VAMI/Applmgmt log (/var/log/vmware/applmgmt/vami.log or /var/log/vmware/applmgmt/applmgmt.log) shows errors similar to below:

2019-12-02T18:48:44.336 [50279]INFO:twisted:"127.0.0.1" - - [02/Dec/2019:10:48:44 +0000] "POST /api HTTP/1.1" 200 2783 "-" "vAPI http client"
2019-12-02T18:50:35.336 [50279]ERROR:vmware.appliance.vapi.auth:Could not parse HOK Token
Traceback (most recent call last):
  File "/usr/lib/applmgmt/vapi/py/vmware/appliance/vapi/auth.py", line 183, in authenticate
    token.validate()
  File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authentication/authentication_sso.py", line 529, in validate
    signing_chain = self.validate_certificate()
  File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authentication/authentication_sso.py", line 700, in validate_certificate
    'One or more certificates cannot be verified.')AuthenticationError: One or more certificates cannot be verified.
2019-12-02T18:50:35.336 [50279]INFO:twisted:"127.0.0.1" - - [02/Dec/2019:10:50:35 +0000] "POST /api HTTP/1.1" 200 339 "-" "vAPI http client"

Environment

VMware vCenter Server 6.5.x
VMware vCenter Server 6.7.x

Cause

This issue will be observed if there are invalid or stale Secure Token Certificate Chains.

Resolution

Note: Take Snapshot of all the vCenter Servers and PSCs (embedded or external) in Enhanced Linked mode before trying below steps.

• Login to vCenter Server using WebClient
• Verify the Signing Certificate Chains (Administration -> Configuration -> Certificates -> STS Signing)
• Delete the stale or invalid certificate chains (for example, Certificates issued by unused / decommissioned PSCs)
• Restart the Services on PSC and VCSA

Following screenshot is a sample list of multiple STS Chains with few stale certificates issued by decommissioned PSCs:

Additional Information

vCenter Server Appliance REST-API Calls Fails With Unable to Authorized User