vpxd service failing to start with Error : "vpxd user does not exist"
Article ID: 336268
Updated On:
Products
VMware vCenter Server
Issue/Introduction
Symptoms:
After restarting the services, the sps service gets stuck while starting as the vpxd service crashes.
- vpxd.log
2020-05-15T16:27:45.328Z error vpxd[7F2F43D7A700] [Originator@6876 sub=vmomi.soapStub[9]] initial service state request failed, disabling pings. error=HTTP Status:400 'Bad Request'
2020-05-15T16:27:45.328Z warning vpxd[7F2F43D7A700] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3
2020-05-15T16:27:45.339Z error vpxd[7F2F7639E800] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\Administrator"
2020-05-15T16:27:45.339Z error vpxd[7F2F7639E800] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\zabbix"
2020-05-15T16:27:45.342Z info vpxd[7F2F7639E800] [Originator@6876 sub=[SSO][AdminAdapter]] [LazyInitAdmin] Initializing
2020-05-15T16:27:45.342Z info vpxd[7F2F7639E800] [Originator@6876 sub=[SSO][AdminAdapter]] [InitSsoAdminServices]
2020-05-15T16:27:45.342Z info vpxd[7F2F7639E800] [Originator@6876 sub=[SSO][AdminAdapter]] Try to connect to SSO VMOMI endpoint
2020-05-15T16:27:45.363Z info vpxd[7F2F7639E800] [Originator@6876 sub=[SSO][AdminAdapter]] [InitSsoAdminServices] successful.
2020-05-15T16:27:45.363Z info vpxd[7F2F7639E800] [Originator@6876 sub=[SSO][SsoAdminFacadeImpl]] [CheckTokenValidity]
2020-05-15T16:27:45.363Z info vpxd[7F2F7639E800] [Originator@6876 sub=[SSO][SsoAdminFacadeImpl]] [CheckTokenValidity] Refreshing SSO token ...
2020-05-15T16:27:45.363Z info vpxd[7F2F7639E800] [Originator@6876 sub=[SSO][SsoAdminFacadeImpl]] [RefreshSsoToken]
2020-05-15T16:27:45.402Z info vpxd[7F2F7639E800] [Originator@6876 sub=[SSO][SsoAdminFacadeImpl]] [RefreshSsoToken] The VC HOK token has been successfully refreshed.
2020-05-15T16:27:45.444Z info vpxd[7F2F7639E800] [Originator@6876 sub=[SSO][AdminAdapter]] Login successful.
2020-05-15T16:27:45.469Z warning vpxd[7F2F7639E800] [Originator@6876 sub=AuthzStorageProvider] [AuthzStorageProvider::SyncClientCache] ignore the exception for adding global permission for user N7Vmacore9Authorize26AuthAlreadyExistsExceptionE(ACE already exists for user 'VSPHERE.LOCAL\Administrator')
--> [context]zKq7AVECAAAAAJ7V6AAQdnB4ZAAA4LArbGlidm1hY29yZS5zbwAAwhcbAB5aGAAtLCAA42sgAAXTIAHrzgFsaWJhdXRoemNsaWVudC5zbwABRtkBAaHaAQGQ3QECDgCidnB4ZAACDDJVAnQ7VQI63lMDkAUCbGliYy5zby42AAJF1lM=[/context]: VSPHERE.LOCAL\Administrator
2020-05-15T16:27:45.470Z error vpxd[7F2F7639E800] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for group "VSPHERE.LOCAL\Administrators"
2020-05-15T16:27:45.470Z error vpxd[7F2F7639E800] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\vsphere-webclient-42fe501d-####-####-####-########45c"
2020-05-15T16:27:45.470Z error vpxd[7F2F7639E800] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\vpxd-42fe501d-####-####-####-########45c"
2020-05-15T16:27:45.470Z error vpxd[7F2F7639E800] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\vpxd-extension-42fe501d-####-####-####-########45c"
2020-05-15T16:27:45.475Z error vpxd[7F2F424C9700] [Originator@6876 sub=vmomi.soapStub[10]] initial service state request failed, disabling pings. error=HTTP Status:400 'Bad Request'
2020-05-15T16:27:45.475Z warning vpxd[7F2F424C9700] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3
2020-05-15T16:27:45.505Z info vpxd[7F2F427CF700] [Originator@6876 sub=[SSO]] [UserDirectorySso] GetUserInfo(VSPHERE.LOCAL\Administrator, false)
2020-05-15T16:27:45.517Z error vpxd[7F2F7639E800] [Originator@6876 sub=Main] vpxd user does not exist
2020-05-15T16:27:45.518Z error vpxd[7F2F7639E800] [Originator@6876 sub=Daemon] Unhandled exception: N5Vmomi5Fault11SystemError9ExceptionE(vmodl.fault.SystemError)
--> [context]zKq7AVECAAAAAJ7V6AAKdnB4ZAAA4LArbGlidm1hY29yZS5zbwAAwhcbAB5aGAHoElV2cHhkAAH9FFUBrzNVAXQ7VQE63lMCkAUCbGliYy5zby42AAFF1lM=[/context]
2020-05-15T16:27:45.518Z info vpxd[7F2F7639E800] [Originator@6876 sub=SupportMgr] Wrote uptime information
2020-05-15T16:27:45.328Z warning vpxd[7F2F43D7A700] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3
2020-05-15T16:27:45.339Z error vpxd[7F2F7639E800] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\Administrator"
2020-05-15T16:27:45.339Z error vpxd[7F2F7639E800] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\zabbix"
2020-05-15T16:27:45.342Z info vpxd[7F2F7639E800] [Originator@6876 sub=[SSO][AdminAdapter]] [LazyInitAdmin] Initializing
2020-05-15T16:27:45.342Z info vpxd[7F2F7639E800] [Originator@6876 sub=[SSO][AdminAdapter]] [InitSsoAdminServices]
2020-05-15T16:27:45.342Z info vpxd[7F2F7639E800] [Originator@6876 sub=[SSO][AdminAdapter]] Try to connect to SSO VMOMI endpoint
2020-05-15T16:27:45.363Z info vpxd[7F2F7639E800] [Originator@6876 sub=[SSO][AdminAdapter]] [InitSsoAdminServices] successful.
2020-05-15T16:27:45.363Z info vpxd[7F2F7639E800] [Originator@6876 sub=[SSO][SsoAdminFacadeImpl]] [CheckTokenValidity]
2020-05-15T16:27:45.363Z info vpxd[7F2F7639E800] [Originator@6876 sub=[SSO][SsoAdminFacadeImpl]] [CheckTokenValidity] Refreshing SSO token ...
2020-05-15T16:27:45.363Z info vpxd[7F2F7639E800] [Originator@6876 sub=[SSO][SsoAdminFacadeImpl]] [RefreshSsoToken]
2020-05-15T16:27:45.402Z info vpxd[7F2F7639E800] [Originator@6876 sub=[SSO][SsoAdminFacadeImpl]] [RefreshSsoToken] The VC HOK token has been successfully refreshed.
2020-05-15T16:27:45.444Z info vpxd[7F2F7639E800] [Originator@6876 sub=[SSO][AdminAdapter]] Login successful.
2020-05-15T16:27:45.469Z warning vpxd[7F2F7639E800] [Originator@6876 sub=AuthzStorageProvider] [AuthzStorageProvider::SyncClientCache] ignore the exception for adding global permission for user N7Vmacore9Authorize26AuthAlreadyExistsExceptionE(ACE already exists for user 'VSPHERE.LOCAL\Administrator')
--> [context]zKq7AVECAAAAAJ7V6AAQdnB4ZAAA4LArbGlidm1hY29yZS5zbwAAwhcbAB5aGAAtLCAA42sgAAXTIAHrzgFsaWJhdXRoemNsaWVudC5zbwABRtkBAaHaAQGQ3QECDgCidnB4ZAACDDJVAnQ7VQI63lMDkAUCbGliYy5zby42AAJF1lM=[/context]: VSPHERE.LOCAL\Administrator
2020-05-15T16:27:45.470Z error vpxd[7F2F7639E800] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for group "VSPHERE.LOCAL\Administrators"
2020-05-15T16:27:45.470Z error vpxd[7F2F7639E800] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\vsphere-webclient-42fe501d-####-####-####-########45c"
2020-05-15T16:27:45.470Z error vpxd[7F2F7639E800] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\vpxd-42fe501d-####-####-####-########45c"
2020-05-15T16:27:45.470Z error vpxd[7F2F7639E800] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\vpxd-extension-42fe501d-####-####-####-########45c"
2020-05-15T16:27:45.475Z error vpxd[7F2F424C9700] [Originator@6876 sub=vmomi.soapStub[10]] initial service state request failed, disabling pings. error=HTTP Status:400 'Bad Request'
2020-05-15T16:27:45.475Z warning vpxd[7F2F424C9700] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3
2020-05-15T16:27:45.505Z info vpxd[7F2F427CF700] [Originator@6876 sub=[SSO]] [UserDirectorySso] GetUserInfo(VSPHERE.LOCAL\Administrator, false)
2020-05-15T16:27:45.517Z error vpxd[7F2F7639E800] [Originator@6876 sub=Main] vpxd user does not exist
2020-05-15T16:27:45.518Z error vpxd[7F2F7639E800] [Originator@6876 sub=Daemon] Unhandled exception: N5Vmomi5Fault11SystemError9ExceptionE(vmodl.fault.SystemError)
--> [context]zKq7AVECAAAAAJ7V6AAKdnB4ZAAA4LArbGlidm1hY29yZS5zbwAAwhcbAB5aGAHoElV2cHhkAAH9FFUBrzNVAXQ7VQE63lMCkAUCbGliYy5zby42AAFF1lM=[/context]
2020-05-15T16:27:45.518Z info vpxd[7F2F7639E800] [Originator@6876 sub=SupportMgr] Wrote uptime information
Environment
VMware vCenter Server 6.5.x
Cause
- The issue is seen when the vpxd solution user is either removed or not found.
- In the etc/passwd file the vpxd user is missing or its name might have been changed.
Resolution
To resolve the issue, make the suggested changes below:
- Take a snapshot of the vcenter without memory
- Log in to the VCSA using SSH.
- Open the /etc/passwd file with editor: vi /etc/passwd
- Press Shift+I to insert.
- Edit the name of the vpxd user or add the missing vpxd user.
- Validate that the file looks like below:
root:x:0:0:root:/root:/bin/appliancesh
bin:x:1:1:bin:/dev/null:/bin/false
daemon:x:6:6:Daemon User:/dev/null:/bin/false
messagebus:x:18:18:D-Bus Message Daemon User:/var/run/dbus:/bin/false
systemd-bus-proxy:x:72:72:systemd Bus Proxy:/:/bin/false
systemd-journal-gateway:x:73:73:systemd Journal Gateway:/:/bin/false
systemd-journal-remote:x:74:74:systemd Journal Remote:/:/bin/false
systemd-journal-upload:x:75:75:systemd Journal Upload:/:/bin/false
systemd-network:x:76:76:systemd Network Management:/:/bin/false
systemd-resolve:x:77:77:systemd Resolver:/:/bin/false
systemd-timesync:x:78:78:systemd Time Synchronization:/:/bin/false
nobody:x:65534:65533:Unprivileged User:/dev/null:/bin/false
sshd:x:50:50:sshd PrivSep:/var/lib/sshd:/bin/false
rpc:x:31:31::/var/lib/rpcbind:/bin/false
tftp:x:999:999:tftp:/home/tftp:/sbin/nologin
apache:x:25:25:Apache Server:/srv/www:/bin/false
ntp:x:87:87:Network Time Protocol:/var/lib/ntp:/bin/false
smmsp:x:26:26:Sendmail Daemon:/dev/null:/bin/false
vapiEndpoint:x:1002:100::/home/vapiEndpoint:/sbin/nologin
sso-user:x:65536:65535::/var/lib/sso-user:/bin/appliancesh
dnsmasq:x:998:998::/home/dnsmasq:/bin/appliancesh
pod:x:1032:59001::/home/pod:/sbin/nologin
cm:x:1001:100::/home/cm:/sbin/nologin
netdumper:x:1015:997:VMware Netdumper User:/var/core/netdumps:/sbin/nologin
vpostgres:x:1012:100::/opt/vmware/vpostgres/current:/sbin/nologin
content-library:x:1006:100::/home/content-library:/sbin/nologin
eam:x:1005:100::/home/eam:/sbin/nologin
imagebuilder:x:1007:100:imagebuilder:/usr/lib/vmware-imagebuilder:/sbin/nologin
mbcs:x:1004:100::/home/mbcs:/sbin/nologin
deploy:x:1019:996:Deploy User:/var/lib/rbd:/sbin/nologin
updatemgr:x:1017:995:Update Manager User:/usr/lib/vmware-updatemgr:/sbin/nologin
vsan-dps:x:1020:100::/home/vsan-dps:/sbin/nologin
vsan-health:x:1008:100::/home/vsan-health:/bin/nologin
vsm:x:1009:100::/home/vsm:/sbin/nologin
vsphere-client:x:1010:100::/home/vsphere-client:/bin/bash
perfcharts:x:1011:100::/home/perfcharts:/sbin/nologin
vsphere-ui:x:1016:100::/home/vsphere-ui:/bin/bash
vpxd:x:1014:59001::/home/vpxd:/bin/false
bin:x:1:1:bin:/dev/null:/bin/false
daemon:x:6:6:Daemon User:/dev/null:/bin/false
messagebus:x:18:18:D-Bus Message Daemon User:/var/run/dbus:/bin/false
systemd-bus-proxy:x:72:72:systemd Bus Proxy:/:/bin/false
systemd-journal-gateway:x:73:73:systemd Journal Gateway:/:/bin/false
systemd-journal-remote:x:74:74:systemd Journal Remote:/:/bin/false
systemd-journal-upload:x:75:75:systemd Journal Upload:/:/bin/false
systemd-network:x:76:76:systemd Network Management:/:/bin/false
systemd-resolve:x:77:77:systemd Resolver:/:/bin/false
systemd-timesync:x:78:78:systemd Time Synchronization:/:/bin/false
nobody:x:65534:65533:Unprivileged User:/dev/null:/bin/false
sshd:x:50:50:sshd PrivSep:/var/lib/sshd:/bin/false
rpc:x:31:31::/var/lib/rpcbind:/bin/false
tftp:x:999:999:tftp:/home/tftp:/sbin/nologin
apache:x:25:25:Apache Server:/srv/www:/bin/false
ntp:x:87:87:Network Time Protocol:/var/lib/ntp:/bin/false
smmsp:x:26:26:Sendmail Daemon:/dev/null:/bin/false
vapiEndpoint:x:1002:100::/home/vapiEndpoint:/sbin/nologin
sso-user:x:65536:65535::/var/lib/sso-user:/bin/appliancesh
dnsmasq:x:998:998::/home/dnsmasq:/bin/appliancesh
pod:x:1032:59001::/home/pod:/sbin/nologin
cm:x:1001:100::/home/cm:/sbin/nologin
netdumper:x:1015:997:VMware Netdumper User:/var/core/netdumps:/sbin/nologin
vpostgres:x:1012:100::/opt/vmware/vpostgres/current:/sbin/nologin
content-library:x:1006:100::/home/content-library:/sbin/nologin
eam:x:1005:100::/home/eam:/sbin/nologin
imagebuilder:x:1007:100:imagebuilder:/usr/lib/vmware-imagebuilder:/sbin/nologin
mbcs:x:1004:100::/home/mbcs:/sbin/nologin
deploy:x:1019:996:Deploy User:/var/lib/rbd:/sbin/nologin
updatemgr:x:1017:995:Update Manager User:/usr/lib/vmware-updatemgr:/sbin/nologin
vsan-dps:x:1020:100::/home/vsan-dps:/sbin/nologin
vsan-health:x:1008:100::/home/vsan-health:/bin/nologin
vsm:x:1009:100::/home/vsm:/sbin/nologin
vsphere-client:x:1010:100::/home/vsphere-client:/bin/bash
perfcharts:x:1011:100::/home/perfcharts:/sbin/nologin
vsphere-ui:x:1016:100::/home/vsphere-ui:/bin/bash
vpxd:x:1014:59001::/home/vpxd:/bin/false
After the file is validated, restart the vCenter services.
Additional Information
Impact/Risks:
vpxd service keeps crashing and user is unable to login to the vCenter.
vpxd service keeps crashing and user is unable to login to the vCenter.
Feedback
Yes
No
Powered by
