Cannot place host in MM mode "Failed to enter namespaces maintenance mode"
search cancel

Cannot place host in MM mode "Failed to enter namespaces maintenance mode"

book

Article ID: 344932

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Unable to place ESXi host in maintenance mode, showing the following general error in the UI:

Failed to enter namespaces maintenance mode due to Error: com.vmware.vapi.std.errors.unauthenticated Messages: vapi.security.authentication.invalid<Unable to authenticate user>


The following excerpts will be in vpxd.log

2023-12-15T09:01:54.487+02:00 info vpxd[00706] [Originator@6876 sub=MoHost opID=lq6a4ce8-542-auto-f3-h5:70000122-90] WCP enterMaintenanceMode vAPI returns error: Error:
-->    com.vmware.vapi.std.errors.unauthenticated
--> Messages:
-->    vapi.security.authentication.invalid<Unable to authenticate user>
-->

wcp.log shows the following:

2023-12-15T13:48:21.362Z warning wcp [vcrestlib/helper.go:195] [opID=hok-auth-handler] Request to service failed; POST, url: https://vcenter_FQDN:443/rest/vcenter/tokenservice/token-exchange, Code: 404, Body: '{"majorErrorCode":404,"name":"com.vmware.vapi.rest.httpNotFound","localizableMessages":[{"args":[],"defaultMessage":"Not found.","id":"com.vmware.vapi.rest.httpNotFound"}],"type":"com.vmware.vapi.std.errors.not_found","value":{"messages":[{"args":[],"default_message":"Not found.","id":"com.vmware.vapi.rest.httpNotFound"}]}}'
2023-12-15T13:48:21.362Z error wcp [tes/client.go:78] [opID=hok-auth-handler] Request to TES failed; token: '<saml2:Assertion xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion" xmlns="urn:oasis:names:tc:SAML:2.0:assertion"

 

Endpoint.log shows the following:

2023-12-15T15:43:33.126+02:00 | ERROR | state-manager1      | MetadataMonitor        | Failed to register metadata 6c498efc-2da2-4fe4-9181-fc131468682d\com.vmware.cis.cls.vapi at http://vcenter
_fqdn:80/cls/
com.vmware.vapi.std.errors.NotFound: NotFound (com.vmware.vapi.std.errors.not_found) => {
  messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => {
  id = datasource.spec.filenotfound,
  defaultMessage = The specified filename '/etc/vmware-content-library/metadata/vdcs/com.vmware.vcenter_fqdn.inventory_metamodel.json' does not exist,
  args = [/etc/vmware-content-library/metadata/vdcs/com.vmware.vcenter_fqdn.inventory_metamodel.json],
  params = <null>,
  localized = <null>
}],
  data = <null>,
  errorType = NOT_FOUND
}
    at com.vmware.vapi.std.errors.NotFound._newInstance2(NotFound.java:182) ~[vapi-runtime.jar:?]
    at sun.reflect.GeneratedMethodAccessor393.invoke(Unknown Source) ~[?:?]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_345]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_345]
    at com.vmware.vapi.internal.bindings.convert.impl.JavaClassStructConverter.createStructBinding(JavaClassStructConverter.java:275) ~[vapi-runtime.jar:?]
    at com.vmware.vapi.internal.bindings.convert.impl.JavaClassStructConverter.fromValue(JavaClassStructConverter.java:79) ~[vapi-runtime.jar:?]
    at com.vmware.vapi.internal.bindings.convert.impl.JavaClassStructConverter.fromValue(JavaClassStructConverter.java:33) ~[vapi-runtime.jar:?]
    at com.vmware.vapi.internal.bindings.TypeConverterImpl$ValueToJavaVisitor.visit(TypeConverterImpl.java:330) ~[vapi-runtime.jar:?]
    at com.vmware.vapi.bindings.type.ErrorType.accept(ErrorType.java:31) ~[vapi-runtime.jar:?]
    at com.vmware.vapi.internal.bindings.TypeConverterImpl.convertToJava(TypeConverterImpl.java:713) ~[vapi-runtime.jar:?]
    at com.vmware.vapi.internal.bindings.Stub.convert(Stub.java:454) ~[vapi-runtime.jar:?]
    at com.vmware.vapi.internal.bindings.Stub.convertError(Stub.java:471) ~[vapi-runtime.jar:?]
    at com.vmware.vapi.internal.bindings.ResultTranslatingHandle.setResult(ResultTranslatingHandle.java:42) ~[vapi-runtime.jar:?]
    at com.vmware.vapi.internal.bindings.ResultTranslatingHandle.setResult(ResultTranslatingHandle.java:21) ~[vapi-runtime.jar:?]
    at com.vmware.vapi.provider.local.LocalProvider$AsyncHandleAdapter.setResult(LocalProvider.java:481) ~[vapi-runtime.jar:?]
    at com.vmware.vapi.provider.local.LocalProvider$AsyncHandleAdapter.setResult(LocalProvider.java:440) ~[vapi-runtime.jar:?]
    at com.vmware.vapi.internal.util.async.StrictAsyncHandle.setResult(StrictAsyncHandle.java:51) ~[vapi-runtime.jar:?]
    at com.vmware.vapi.internal.bindings.ApiMethodSkeleton$ValidatingAsyncHandle.setResult(ApiMethodSkeleton.java:546) ~[vapi-runtime.jar:?]
    at com.vmware.vapi.internal.bindings.ApiMethodSkeleton$ValidatingAsyncHandle.setResult(ApiMethodSkeleton.java:500) ~[vapi-runtime.jar:?]
    at com.vmware.vapi.internal.bindings.server.impl.AsyncContextImpl.setError(AsyncContextImpl.java:97) ~[vapi-runtime.jar:?]
    at com.vmware.vapi.internal.metadata.service.metamodel.SourceProviderImpl.create(SourceProviderImpl.java:75) ~[vapi-metadata-service.jar:?]
    at com.vmware.vapi.metadata.metamodel.SourceApiInterface$CreateApiMethod.doInvoke(SourceApiInterface.java:44) ~[vapi-common.jar:?]
    at com.vmware.vapi.internal.bindings.ApiMethodSkeleton.invoke(ApiMethodSkeleton.java:232) ~[vapi-runtime.jar:?]
    at com.vmware.vapi.provider.ApiMethodBasedApiInterface.invoke(ApiMethodBasedApiInterface.java:86) ~[vapi-runtime.jar:?]
    at com.vmware.vapi.provider.local.LocalProvider.invokeMethodInt(LocalProvider.java:399) ~[vapi-runtime.jar:?]
    at com.vmware.vapi.provider.local.LocalProvider.invoke(LocalProvider.java:270) ~[vapi-runtime.jar:?]
    at com.vmware.vapi.internal.bindings.Stub.invoke(Stub.java:241) ~[vapi-runtime.jar:?]
    at com.vmware.vapi.internal.bindings.Stub.invokeMethodAsync(Stub.java:191) ~[vapi-runtime.jar:?]
    at com.vmware.vapi.internal.bindings.Stub.invokeMethod(Stub.java:137) ~[vapi-runtime.jar:?]
    at com.vmware.vapi.metadata.metamodel.SourceStub.create(SourceStub.java:47) ~[vapi-common-client.jar:?]
    at com.vmware.vapi.metadata.metamodel.SourceStub.create(SourceStub.java:37) ~[vapi-common-client.jar:?]
    at com.vmware.vapi.endpoint.metadata.util.MetadataHelperFactory$1.addFile(MetadataHelperFactory.java:55) ~[vapi-endpoint-1.0.0.jar:?]
    at com.vmware.vapi.endpoint.metadata.util.MetadataOperationsHelper.addFile(MetadataOperationsHelper.java:45) ~[vapi-endpoint-1.0.0.jar:?]
    at com.vmware.vapi.endpoint.metadata.util.MetadataMonitor.registerService(MetadataMonitor.java:119) ~[vapi-endpoint-1.0.0.jar:?]
    at com.vmware.vapi.endpoint.metadata.util.MetadataMonitor.registerAllNew(MetadataMonitor.java:76) [vapi-endpoint-1.0.0.jar:?]
    at com.vmware.vapi.endpoint.metadata.util.MetadataMonitor.sync(MetadataMonitor.java:58) [vapi-endpoint-1.0.0.jar:?]
    at com.vmware.vapi.endpoint.api.MetadataSynchronizationBuilder.syncMetadata(MetadataSynchronizationBuilder.java:177) [vapi-endpoint-1.0.0.jar:?]
    at com.vmware.vapi.endpoint.api.MetadataSynchronizationBuilder.rebuild(MetadataSynchronizationBuilder.java:141) [vapi-endpoint-1.0.0.jar:?]
at com.vmware.vapi.state.impl.DefaultStateManager.rebuild(DefaultStateManager.java:405) [vapi-endpoint-1.0.0.jar:?]
    at com.vmware.vapi.state.impl.DefaultStateManager$2.doReconfig(DefaultStateManager.java:443) [vapi-endpoint-1.0.0.jar:?]
    at com.vmware.vapi.state.impl.DefaultStateManager$2.run(DefaultStateManager.java:432) [vapi-endpoint-1.0.0.jar:?]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_345]
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_345]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_345]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_345]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_345]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_345]
    at java.lang.Thread.run(Thread.java:750) [?:1.8.0_345]

 

Environment

VMware vSphere 7.0.x
VMware vSphere ESXi 7.0

Cause

Issue is caused by underlying problem with the service registrations on the vCenter.

Resolution

Using lsdoctor -r option followed by number 2 "Replace all services with new services."

Using the 'lsdoctor' Tool

Additional Information

Impact/Risks:

Unable to place host into maintenance mode.

Maintenance mode still possible from ESXi UI.

Powered by Wolken Software