"Failed to exit namespaces from maintenance mode due to Error: com.vmware.vapi.std.errors.internal_server_error", exit maintenance mode operation fails at 95%

Products

• VMware vCenter Server

Issue/Introduction

• Exit maintenance mode operation on ESXi host continually fails at 95% with below error message : 
  
  Failed to exit namespaces from maintenance mode due to Error: com.vmware.vapi.std.errors.internal_server_error Messages: vcenter.wcp.authorization.general<An unexpected error occurred during authorization.> . Retry 46

• WCP service log (/var/log/vmware/wcpsvc.log) shows ServiceUnavailable errors while trying to check the privileges.
  
  YYYY-MM-DDTHH:MM:SS.530Z warning wcp [vcrestlib/helper.go:176] [opID=wcp-AuthzFilter] Request to service failed; POST, url: http://localhost:1080/rest/com/vmware/cis/authz/privilege?~action=batch-has-privileges, Code: 503, Body: '{"type":"com.vmware.vapi.std.errors.service_unavailable","value":{"error_type":"SERVICE_UNAVAILABLE","messages":[{"args":[],"default_message":"Service unavailable.","id":"com.vmware.vapi.endpoint.cis.ServiceUnavailable"}]}}'
YYYY-MM-DDTHH:MM:SS.530Z error wcp [namespace/authz.go:201] [opID=wcp-AuthzFilter] Failed to check privileges for user:  vsphere.local\vpxd-<machineid>, groupnames: [[e-mail] [e-mail] [e-mail] [e-mail] [e-mail] [e-mail] [e-mail]], resources: [{PermissionFolder global-permission}], privileges: [System.Anonymous]: HTTP request failed; POST, url: http://localhost:1080/rest/com/vmware/cis/authz/privilege?~action=batch-has-privileges, code: 503, body: '{"type":"com.vmware.vapi.std.errors.service_unavailable","value":{"error_type":"SERVICE_UNAVAILABLE","messages":[{"args":[],"default_message":"Service unavailable.","id":"com.vmware.vapi.endpoint.cis.ServiceUnavailable"}]}}'
YYYY-MM-DDTHH:MM:SS.530Z warning wcp [vcrestlib/helper.go:176] [opID=wcp-AuthzFilter] Request to service failed; POST, url: http://localhost:1080/rest/com/vmware/cis/authz/privilege?~action=batch-has-privileges, Code: 503, Body: '{"type":"com.vmware.vapi.std.errors.service_unavailable","value":{"error_type":"SERVICE_UNAVAILABLE","messages":[{"args":[],"default_message":"Service unavailable.","id":"com.vmware.vapi.endpoint.cis.ServiceUnavailable"}]}}'
  
  
• Checking the running service status shows vpxd-svcs in stopped state along with some other services
  
  Running:
 applmgmt lookupsvc lwsmd observability observability-vapi pschealth vc-ws1a-broker vlcm vmafdd vmcad vmdird vmware-analytics vmware-cis-license vmware-content-library vmware-eam vmware-envoy vmware-envoy-hgw vmware-envoy-sidecar vmware-infraprofile vmware-perfcharts vmware-pod vmware-postgres-archiver vmware-rhttpproxy vmware-sca vmware-stsd vmware-trustmanagement vmware-updatemgr vmware-vapi-endpoint vmware-vdtc vmware-vmon vmware-vpostgres vmware-vpxd vmware-vsm vsphere-ui vtsdb wcp
Stopped:
 vmcam vmonapi vmware-certificateauthority vmware-certificatemanagement vmware-hvc vmware-imagebuilder vmware-netdumper vmware-rbd-watchdog vmware-sps vmware-topologysvc vmware-vcha vmware-vpxd-svcs vmware-vsan-health vstats
  
  
• vMon logs (/var/log/vmware/vmon/vmon.log) shows pre-start errors while trying to start vpxd-svcs service
  
  YYYY-MM-DDTHH:MM:SS.150Z Wa(03) host-2611 <vpxd-svcs> Service pre-start command's stderr: Traceback (most recent call last):
YYYY-MM-DDTHH:MM:SS.150Z Wa(03)+ host-2611   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py", line 213, in <module>
YYYY-MM-DDTHH:MM:SS.150Z Wa(03)+ host-2611     endpoint_registration_runner()
YYYY-MM-DDTHH:MM:SS.150Z Wa(03)+ host-2611   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py", line 99, in endpoint_registration_runner
YYYY-MM-DDTHH:MM:SS.150Z Wa(03)+ host-2611     UpdateTaggingServiceGrpcEndpoint(logger).run()
YYYY-MM-DDTHH:MM:SS.150Z Wa(03)+ host-2611   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/tagging_grpc_registration.py", line 51, in run
YYYY-MM-DDTHH:MM:SS.150Z Wa(03)+ host-2611
YYYY-MM-DDTHH:MM:SS.163Z Wa(03) host-2611 <vpxd-svcs> Service pre-start command's stderr:     self.update_endpoints()
YYYY-MM-DDTHH:MM:SS.163Z Wa(03)+ host-2611   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/tagging_grpc_registration.py", line 80, in update_endpoints
YYYY-MM-DDTHH:MM:SS.163Z Wa(03)+ host-2611     ls_obj = LookupServiceClient(ls_url, retry_count=5)
YYYY-MM-DDTHH:MM:SS.163Z Wa(03)+ host-2611   File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 316, in __init__
YYYY-MM-DDTHH:MM:SS.163Z Wa(03)+ host-2611     self._init_service_content()
YYYY-MM-DDTHH:MM:SS.163Z Wa(03)+ host-2611   File "/usr/lib/python3.10/http/client.py", line 976, in send
YYYY-MM-DDTHH:MM:SS.163Z Wa(03)+ host-2611     self.connect()
YYYY-MM-DDTHH:MM:SS.163Z Wa(03)+ host-2611   File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1160, in connect
YYYY-MM-DDTHH:MM:SS.163Z Wa(03)+ host-2611     six.moves.http_client.HTTPSConnection.connect(self)
YYYY-MM-DDTHH:MM:SS.163Z Wa(03)+ host-2611   File "/usr/lib/python3.10/ssl.py", line 1100, in _create
YYYY-MM-DDTHH:MM:SS.163Z Wa(03)+ host-2611     self.do_handshake()
YYYY-MM-DDTHH:MM:SS.163Z Wa(03)+ host-2611   File "/usr/lib/python3.10/ssl.py", line 1371, in do_handshake
YYYY-MM-DDTHH:MM:SS.163Z Wa(03)+ host-2611     self._sslobj.do_handshake()
YYYY-MM-DDTHH:MM:SS.163Z Wa(03)+ host-2611 ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1007)
YYYY-MM-DDTHH:MM:SS.163Z Wa(03)+ host-2611
YYYY-MM-DDTHH:MM:SS.243Z Er(02) host-2611 <vpxd-svcs> Service pre-start command failed with exit code 1.

Cause

This issue is caused when the MACHINE_SSL_CERT certificate is expired on vCenter Server.

Resolution

Replace the expired Machine SSL Certificate with VMCA signed certificate by following the instructions in Replacing the vSphere 6.x Machine SSL certificate with a VMware Certificate Authority issued certificate.