vpxd crashes intermittently with "Signature verification error. No verification key available."

Products

• VMware vCenter Server 8.0
• VMware SDDC Manager

Issue/Introduction

• vpxd service crashes randomly and does not create a core dump every time.
• Manually starting vpxd service via VAMI or CLI works.
• Below log entries can be seen in the vpxd.log:
  
  
  • A client attempts to re-login with already authenticated session:
    
    YYYY-MM-DDThh:mm:ss.886-05:00 warning vpxd[1408119] [Originator@6876 sub=User opID= ####] [VpxdUser] Attempt to re-register session with cnxId=#####-####-###-###-###### failed
    
    
  • Session is authenticated for the first time here:
    
    YYYY-MM-DDThh:mm:ss.901-05:00 info vpxd[1407974] [Originator@6876 sub=vpxLro opID= ####] [VpxLRO] -- BEGIN lro-364089 -- SessionManager -- vim.SessionManager.login -- 521###c0-####-e351-###-bdff6d###abd
    
    
  • And then client started to call SessionManager.login() again and again:
    
    YYYY-MM-DDThh:mm:ss.026-05:00 info vpxd[1408272] [Originator@6876 sub=vpxLro opID= ####] [VpxLRO] -- BEGIN lro-364651 -- SessionManager -- vim.SessionManager.login -- 521###c0-####-e351-###-bdff6d###abd(52####da-a##8-e98b-7b7b-########)
YYYY-MM-DDT04:13:47.340-05:00 warning vpxd[1408272] [Originator@6876 sub=User opID= #####] [VpxdUser] Attempt to re-register session with cnxId=######-4###a-e###1-a285-###### failed
    
    
  • Based on following log line, the token is successfully acquired.
    
    YYYY-MM-DDThh:mm:ss.340-05:00 info vpxd[1408272] [Originator@6876 sub=SsoClient opID= ####] Successfully acquired token: 
    
    
  • However, LoginByToken() fails as vpxd can't validate signature:
    
    YYYY-MM-DDThh:mm:ss.244-05:00 error vpxd[1408038] [Originator@6876 sub=VMOMI opID= ###############] SAML token validation failed. Error: N9SsoClient25InvalidSignatureExceptionE(Signature verification error. No verification key available.)
  • This is a broken SAML token which vpxd can't validate:
    
    YYYY-MM-DDThh:mm:ss.536Z INFO sts[63:tomcat-http--25] [CorId=06###898-6c81-####-###-#####] [com.vmware.identity.token.impl.X509TrustChainKeySelector] Failed to find trusted path to signing certificate <CN=ssoserverSign>

Environment

• VMware vCenter Server 8.0

Cause

• The SAML token is invalid and vpxd is unable to validate it.

Resolution

• This is a known issue and is resolved in the latest patch release for 8.0U3 P05 build number 24674346.

Workaround:

Note: Take a snapshot of the vCenter before following the below steps. If the vCenters are in ELM, please take a powered off snapshot of all the nodes.

• Set the advanced parameter in vCenter UI. This does not require to restart vpxd service:
  
  • Go to vCenter -> Configure -> Advanced Settings -> Edit Settings --> Add a new parameter as below:
    
    • Name: config.vpxd.authorize.sessionCanOutliveToken
      Value: true
      
      Note: These values are case sensitive. Please ensure that the cases are correct as above and there are no extra spaces 

OR

• In vpxd.cfg set vpxd.authorize.sessionCanOutliveToken to true.These steps require restarting vpxd service:
   
  • SSH to the vCenter.
  • Browse to /etc/vmware-vpx directory: cd /etc/vmware-vpx
  • Take a backup of the vpxd.cfg file: cp vpxd.cfg vpxd.cfg.bkp
  • Open vpxd.cfg in vi editor: vi vpxd.cfg
  • Locate the tag <vpxd>.
  • Add the below entries below the <vpxd> tag:
     <authorize>
      <sessionCanOutliveToken>true</sessionCanOutliveToken>
  </authorize>
  • Hit ESC key and hit colon ':' and type wq! to save and exit.
  • Restart vpxd service: service-control --restart vpxd