Host fails to enter into maintenance mode due to the following error : "Failed to enter namespaces maintenance mode due to Error : system_error Messages: vapi.invalid.result.code<Recv of frame failed with code: 503 Service Unavailable> . Retry 1"

Products

• VMware vCenter Server

Issue/Introduction

ESXi host fails to enter Maintenance Mode and remains stuck at 2% progress. During the operation, the task eventually fails with the following error:

Failed to enter namespaces maintenance mode due to Error: system_error
Messages: vapi.invalid.result.code <Recv of frame failed with code: 503 Service Unavailable>
Retry 1

The following may be seen in /var/log/vmware/vpxd/vpxd.log:

YYYY-MM-DDTHH:MM:SS error vpxd[07040] [Originator@6876 sub=MoHost opID=########### : ####### - ##] Failed to set VAPI security context; N4Vpxd7Clients29BadVapiConfigurationExceptionE(Error:
system_error
-> Messages:
vapi. invalid. result. code<Recv of frame failed with code: 503 Service Unavailable>

YYYY-MM-DDTHH:MM:SS warning vpxd[07077] [Originator@6876 sub=MoHost opID=########## : ####### - ##] VAPI provider failed; <last binding: << TCP '127.0.0.1 : 54156'>, <TCP '127.0.0.1 : 1080'>> >
system_error
-> Messages:
vapi. invalid. result. code<Recv of frame failed with code: 503 Service Unavailable>

YYYY-MM-DDTHH:MM:SS warning vpxd[07040] [Originator@6876 sub=MoHost opID=########## : ####### - ##] Failed request to VAPI service; Error:
system_error
-> Messages:
vapi. invalid. result. code<Recv of frame failed with code: 503 Service Unavailable>

YYYY-MM-DDTHH:MM:SS error vpxd[07040] [Originator@6876 sub=MoHost opID=########## : ####### - ##] WCP enterMaintenanceMode vAPI returns error: Error:
system_error
-> Messages:
vapi. invalid. result. code<Recv of frame failed with code: 503 Service Unavailable>

Environment

VCF 9.0

Cause

When the service status was checked on the vCenter Server, it was observed that the WCP  and vmware-percharts services were stopped.

The failure to enter maintenance mode occurs due to unavailability of VAPI services required for namespace operations, caused by the WCP  and vmware-percharts services being stopped on the vCenter Server.

When these services are not running, vCenter is unable to establish a valid VAPI security context or communicate with the local VAPI provider. As a result, namespace-related maintenance mode requests fail

Resolution

The issue was resolved by starting the required services on the vCenter Server.

Log in to the vCenter Server using SSH.

Start the stopped services by running the following commands:

service-control --start wcp
service-control --start vmware-perfcharts

After the services are running, retry placing the ESXi host into Maintenance Mode.

Once the WCP and vmware-perfcharts services were started, the ESXi host was able to successfully enter maintenance mode without any errors.

Additional Information

https://knowledge.broadcom.com/external/article/340837/wcp-service-fails-to-start-during-vcente.html

https://knowledge.broadcom.com/external/article/326225/wcp-service-fails-to-start-in-vcenter-se.html

https://knowledge.broadcom.com/external/article/383389/vcenter-service-vmwareperfcharts-failing.html

https://knowledge.broadcom.com/external/article/326213/vmwareperfcharts-service-failed-to-start.html