Vpxd Exception: Error while doing login to VPXD service, SPS service fails to start

Products

• VMware vCenter Server

Issue/Introduction

• VMware vSphere Profile-Driven Storage Service and VMware Update Manager service failing to start
• In the /var/log/sps/sps.log, you see message similar to:
  

  [main] INFO  opId=############ com.vmware.identity.token.impl.Util - Reading resources from zip file path=[/usr/lib/vmware-vpx/sps/lib/samltoken-1.0.jar]
  [main] INFO  opId=############ com.vmware.identity.token.impl.Util - Reading resources from decoded zip file path=[/usr/lib/vmware-vpx/sps/lib/samltoken-1.0.jar]
  [main] INFO  opId=############ com.vmware.identity.token.impl.SamlTokenImpl - SAML token for SubjectNameId [value=vpxd-extension-xxxxxxxxxxxxxxxxxxxxxxxxx
  [e-mail], format=http://schemas.xmlsoap.org/claims/UPN] successfully parsed from Element
  [main] INFO  opId=############ com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl - Successfully acquired token for user: {Name: vpxd-extension-xxxxxxxxxxxxxxxxxxxxxxxxx, Domain: vsphere.local}
  [main] INFO  opId=############ com.vmware.vim.storage.common.identity.ServiceSolutionUserByHoKToken - Service solution user token acquired successfully. Expiration XXXXXXXXXXXX
  [main] INFO  opId=############ com.vmware.vim.storage.common.identity.ServiceSolutionUserByHoKToken - SPS solution user initialized successfully
  [main] ERROR opId=############ com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientImpl - VPXD client login failed.
  [main] ERROR opId=############ com.vmware.vim.storage.common.task.retry.CallableRetryDecorator - Caught exception -
  com.vmware.vim.storage.common.serviceclient.vpxd.VpxdException: Error while doing login to VPXD service
          at com.vmware.vim.storage.common.serviceclient.vpxd.VpxdException.fromEx(VpxdException.java:53)
          at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientImpl.loginByToken(VpxdClientImpl.java:159)
          at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientLifeCycle.login(VpxdClientLifeCycle.java:129)
          at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientLifeCycle.login(VpxdClientLifeCycle.java:34)
          at com.vmware.vim.storage.common.serviceclient.ConnectionInitializationTask$CallableTemplate.call(ConnectionInitializationTask.java:118)
          at com.vmware.vim.storage.common.task.retry.CallableRetryDecorator.call(CallableRetryDecorator.java:64)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientManagerImpl.initialize(VpxdClientManagerImpl.java:106)
          at com.vmware.sps.StorageMain.commonInitialization(StorageMain.java:188)
          at com.vmware.sps.StorageMain.main(StorageMain.java:67)
  Caused by: java.util.concurrent.TimeoutException
          at com.vmware.vim.vmomi.core.impl.BlockingFuture.get(BlockingFuture.java:102)
          at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientImpl.loginByToken(VpxdClientImpl.java:153)
          ... 8 more
  [main] ERROR opId=############ com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientImpl - VPXD client login failed.

 

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

 

Environment

VMware vCenter Server 7.0.x
VMware vCenter Server Appliance 6.7.x

Cause

This issue is caused when the SPS detects VPXD is busy for more than 30 seconds.

Resolution

This issue is fixed in vCenter Server 7.0, available at vCenter Server Update and Patch Releases .


Workaround:

1. Take snapshot of vCenter VM (offline snapshot of all vCenters if in ELM)
2. Log in to the vCenter Server Appliance via ssh as root user
3. Navigate to conf using below command
   

   cd /usr/lib/vmware-vpx/sps/conf/

4. Backup the common.properties file to a different location
   

   cp /etc/vmware/common.properties /tmp/backup/common.properties

5. Use vi editor and add the below line to the same file ( Open the file using the vi editor with the command vi <filename>. Once the file is open, press Esc, then press i to begin editing)
   

   vpxdClient.waitDurationInSec=1800

6. save and exit the file ( press "Shift + :", then type wq! and hit Enter. )
7. Restart the SPS service using the below command
   

   vmon-cli -r sps

8. Once the service is restarted then restart rest of the services
   

   service-control --stop --all && service-control --start --all