Error while placing host in Maintenance Mode via vCenter Server "Host cannot enter maintenance mode because it failed to enter Namespaces maintenance mode on the node".
search cancel

Error while placing host in Maintenance Mode via vCenter Server "Host cannot enter maintenance mode because it failed to enter Namespaces maintenance mode on the node".

book

Article ID: 322351

calendar_today

Updated On:

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

The Error displayed on the vSphere web-client  "Host cannot enter maintenance mode because it failed to enter Namespaces maintenance mode on the node" 

Messages similar to these are seen in /var/log/vmware/vpxd.log


YYYY-MM-DDTHH:MM:SSZ info vpxd[24862] [Originator@6876 sub=vpxLro opID=kk8e2pzk-1583-auto-181-h5:70000604-6] [VpxLRO] -- BEGIN task-89025 -- host-142535 -- vim.HostSystem.enterMaintenanceMode -- 527198e2-78a8-6adf-2f24-5733abcde5f1(5223a236-873e-af77-5a06-0796773dcb8f)
YYYY-MM-DDTHH:MM:SSZ info vpxd[24791] [Originator@6876 sub=vpxLro opID=kk8e2pzk-1583-auto-181-h5:70000604-6-e0] [VpxLRO] -- BEGIN lro-19016 -- ServiceInstance -- vim.ServiceInstance.retrieveContent -- 5210fb6b-c2f9-2840-91cc-e3ef4ff30c2f
YYYY-MM-DDTHH:MM:SSZ info vpxd[24791] [Originator@6876 sub=vpxLro opID=kk8e2pzk-1583-auto-181-h5:70000604-6-e0] [VpxLRO] -- FINISH lro-19016
YYYY-MM-DDTHH:MM:SSZ info vpxd[24795] [Originator@6876 sub=vpxLro opID=kk8e2pzk-1583-auto-181-h5:70000604-6-f3] [VpxLRO] -- BEGIN lro-19025 -- SessionManager -- vim.SessionManager.loginByToken -- 52022f3c-725f-e9b2-37b2-1701bbecbb98
YYYY-MM-DDTHH:MM:SSZ info vpxd[24795] [Originator@6876 sub=[SSO] opID=kk8e2pzk-1583-auto-181-h5:70000604-6-f3] [UserDirectorySso] GetUserInfo(vsphere.local\vpxd-5c8f08b0-79ae-4895-aee1-d09555aa3ebe, false)
YYYY-MM-DDTHH:MM:SSZ info vpxd[24795] [Originator@6876 sub=[SSO] opID=kk8e2pzk-1583-auto-181-h5:70000604-6-f3] [UserDirectorySso] GetUserInfo(vsphere.local\vpxd-5c8f08b0-79ae-4895-aee1-d09555aa3ebe, false) res: VSPHERE.LOCAL\vpxd-5c8f08b0-79ae-4895-aee1-d09555aa3ebe
YYYY-MM-DDTHH:MM:SSZ info vpxd[24795] [Originator@6876 sub=AuthorizeManager opID=kk8e2pzk-1583-auto-181-h5:70000604-6-f3] [Auth]: User VSPHERE.LOCAL\vpxd-5c8f08b0-79ae-4895-aee1-d09555aa3ebe
YYYY-MM-DDTHH:MM:SSZ warning vpxd[24795] [Originator@6876 sub=AuthorizeManager opID=kk8e2pzk-1583-auto-181-h5:70000604-6-f3] Refresh function is not configured.User data can't be added to scheduler.User name: VSPHERE.LOCAL\vpxd-5c8f08b0-79ae-4895-aee1-d09555aa3ebe
YYYY-MM-DDTHH:MM:SSZ info vpxd[24795] [Originator@6876 sub=User opID=kk8e2pzk-1583-auto-181-h5:70000604-6-f3] SSO Login > User: 'VSPHERE.LOCAL\vpxd-5c8f08b0-79ae-4895-aee1-d09555aa3ebe', Groups: '{Name: SolutionUsers; Domain:vsphere.local} {Name: ActAsUsers; Domain:vsphere.local} {Name: Users; Domain:vsphere.local} {Name: ComponentManager.Administrators; Domain:vsphere.local} {Name: LicenseService.Administrators; Domain:vsphere.local} {Name: SystemConfiguration.Administrators; Domain:vsphere.local} {Name: Everyone; Domain:vsphere.local} ', DelegationChain: ''
YYYY-MM-DDTHH:MM:SSZ info vpxd[24795] [Originator@6876 sub=vpxLro opID=kk8e2pzk-1583-auto-181-h5:70000604-6-f3] [VpxLRO] -- FINISH lro-19025
YYYY-MM-DDTHH:MM:SSZ info vpxd[24791] [Originator@6876 sub=vpxLro opID=kk8e2pzk-1583-auto-181-h5:70000604-6-daf8-f] [VpxLRO] -- BEGIN lro-19028 -- ServiceInstance -- vim.ServiceInstance.retrieveContent -- 52e0678e-56f0-a6bb-ea7e-75973a51a676
YYYY-MM-DDTHH:MM:SSZ info vpxd[24791] [Originator@6876 sub=vpxLro opID=kk8e2pzk-1583-auto-181-h5:70000604-6-daf8-f] [VpxLRO] -- FINISH lro-19028
YYYY-MM-DDTHH:MM:SSZ info vpxd[27618] [Originator@6876 sub=vpxLro opID=kk8e2pzk-1583-auto-181-h5:70000604-6-daf8-4] [VpxLRO] -- BEGIN lro-19030 -- TaskManager -- vim.TaskManager.createTask -- 52eb6d4e-9442-124a-0b9a-08c637e31ab5(5263e969-d5dd-cf61-ac71-3a6489a9d2df)
YYYY-MM-DDTHH:MM:SSZ info vpxd[27618] [Originator@6876 sub=vpxLro opID=kk8e2pzk-1583-auto-181-h5:70000604-6-daf8-4] [VpxLRO] -- FINISH lro-19030
YYYY-MM-DDTHH:MM:SSZ info vpxd[27570] [Originator@6876 sub=vpxLro opID=kk8e2pzk-1583-auto-181-h5:70000604-6-daf8-55] [VpxLRO] -- BEGIN lro-19033 -- session[52fb1b68-6e9d-24f4-27b6-16f92bd0b6c4]521c1e7e-6d24-0fa9-2696-35f808dbcdeb -- vim.view.ListView.modify -- 52fb1b68-6e9d-24f4-27b6-16f92bd0b6c4(52651a3c-70cd-a48e-9347-0d19302b5d45)
YYYY-MM-DDTHH:MM:SSZ info vpxd[27570] [Originator@6876 sub=vpxLro opID=kk8e2pzk-1583-auto-181-h5:70000604-6-daf8-55] [VpxLRO] -- FINISH lro-19033
YYYY-MM-DDTHH:MM:SSZ info vpxd[24862] [Originator@6876 sub=MoHost opID=kk8e2pzk-1583-auto-181-h5:70000604-6] vSAN PerformResourceCheck status: resourceCheckCompleted
YYYY-MM-DDTHH:MM:SSZ info vpxd[24862] [Originator@6876 sub=MoHost opID=kk8e2pzk-1583-auto-181-h5:70000604-6] vSAN PerformResourceCheckResult status: green
YYYY-MM-DDTHH:MM:SSZ info vpxd[24862] [Originator@6876 sub=vpxCrypt opID=kk8e2pzk-1583-auto-181-h5:70000604-6] Failed to read X509 cert; err: 151441516
YYYY-MM-DDTHH:MM:SSZ warning vpxd[24862] [Originator@6876 sub=MoHost opID=kk8e2pzk-1583-auto-181-h5:70000604-6] [Invoke] Host 'localhost' Failed to acquire Session: N4Vpxd7Langley29BadVapiConfigurationExceptionE(Error:
YYYY-MM-DDTHH:MM:SSZ info vpxd[24862] [Originator@6876 sub=MoHost opID=kk8e2pzk-1583-auto-181-h5:70000604-6] WCP enterMaintenanceMode vAPI returns error: Error:
YYYY-MM-DDTHH:MM:SSZ warning vpxd[24862] [Originator@6876 sub=MoHost opID=kk8e2pzk-1583-auto-181-h5:70000604-6] [Close] Host 'localhost' Failed to delete Session: N4Vpxd7Langley29BadVapiConfigurationExceptionE(Error:
YYYY-MM-DDTHH:MM:SSZ info vpxd[24862] [Originator@6876 sub=vpxCrypt opID=kk8e2pzk-1583-auto-181-h5:70000604-6] Failed to read X509 cert; err: 151441516
YYYY-MM-DDTHH:MM:SSZ warning vpxd[24862] [Originator@6876 sub=MoHost opID=kk8e2pzk-1583-auto-181-h5:70000604-6] [Invoke] Host 'localhost' Failed to acquire Session: N4Vpxd7Langley29BadVapiConfigurationExceptionE(Error:
YYYY-MM-DDTHH:MM:SSZ info vpxd[24862] [Originator@6876 sub=MoHost opID=kk8e2pzk-1583-auto-181-h5:70000604-6] WCP exitMaintenanceMode vAPI returns error: Error:
YYYY-MM-DDTHH:MM:SSZ warning vpxd[24862] [Originator@6876 sub=MoHost opID=kk8e2pzk-1583-auto-181-h5:70000604-6] [Close] Host 'localhost' Failed to delete Session: N4Vpxd7Langley29BadVapiConfigurationExceptionE(Error:
YYYY-MM-DDTHH:MM:SSZ info vpxd[24862] [Originator@6876 sub=MoHost opID=kk8e2pzk-1583-auto-181-h5:70000604-6] Set vMotion as evacuation action: 0 for powered VMs on host: [vim.HostSystem:host-142535,nrv-shst12.datacenter.loc]
YYYY-MM-DDTHH:MM:SSZ info vpxd[24862] [Originator@6876 sub=vpxLro opID=kk8e2pzk-1583-auto-181-h5:70000604-6] [VpxLRO] -- FINISH task-89025
YYYY-MM-DDTHH:MM:SSZ info vpxd[24862] [Originator@6876 sub=Default opID=kk8e2pzk-1583-auto-181-h5:70000604-6] [VpxLRO] -- ERROR task-89025 -- host-142535 -- vim.HostSystem.enterMaintenanceMode: vim.fault.InvalidState:


Messages similar to these are seen in /var/log/vmware/wcp/wcpsvc.log

YYYY-MM-DDTHH:MM:SSZ  debug wcp [opID=hok-auth-handler] Getting HOK signer; store: wcp, alias: wcp
YYYY-MM-DDTHH:MM:SSZ  error wcp [opID=hok-auth-handler] STS Issue HOK request failed; err: ns0:RequestFailed: Error occured looking for solution user :: More than one solution user found :: More than one solution user found


Cause

When using enhanced link mode and you replace the solution users certificate using the certificate-manager, with default values a unique SubjectDN is not being generated for wcp solution users.

Resolution

Follow the below steps to resolve the issue.

  1. Make a directory called certs 
    mkdir certs && cd certs
  2. Generate a CSR request file for wcp
    cp /usr/lib/vmware-vmca/share/config/certool.cfg /certs/wcp.cfg
  3. Retrieve the Machine id, hostname for the vCenter Server.
    /usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost && hostname
  4. Update wcp solution user configuration file with desired values

    vi /var/tmp/vmware/wcp.cfg
    Country = 

    Name = wcp-<insert unique machine id>

    Organization = 

    OrgUnit = mID-<insert unique machine id>

    State = 

    Locality = 

    IPAddress = 

    Email = 

    Hostname = <FQDN fetched from output of cmd - hostname>
  5. Generate wcp solution user key
    /usr/lib/vmware-vmca/bin/certool --server localhost --genkey --privkey=/root/wcp.key --pubkey=/root/wcp.pub
  6. Generate wcp solution user certificate
    /usr/lib/vmware-vmca/bin/certool --server=localhost  --gencert --privkey=/root/wcp.key --config=/certs/wcp.cfg --cert=/root/wcp.crt
  7. Get wcp service name using dir-cli ; default name - wcp-<machine id>
    /usr/lib/vmware-vmafd/bin/dir-cli service  list
  8. Update wcp service with new wcp certificate
    /usr/lib/vmware-vmafd/bin/dir-cli service update --name <insert wcp service name from the service list> --cert /root/wcp.crt
  9. Delete wcp solution user entry from vecs store 
    /usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store wcp --alias wcp -y

    /usr/lib/vmware-vmafd/bin/vecs-cli force-refresh
  10. Update wcp solution user certificate to vecs store
    /usr/lib/vmware-vmafd/bin/vecs-cli entry create --store wcp --alias wcp --cert  /root/wcp.crt --key /root/wcp.key
  11. Verify wcp certificate is updated; Subject should contain unique CN as updated in wcp.cfg
    /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store wcp --alias wcp --text
  12. Restart services
    service-control --stop --all && service-control --start --all && service-control --status
  13. Proceed with placing ESXi host in Maintenance Mode

Powered by Wolken Software