vSphere Supervisor Installation stuck in configuring and getting the error "waiting for master VM to be provisioned"
search cancel

vSphere Supervisor Installation stuck in configuring and getting the error "waiting for master VM to be provisioned"

book

Article ID: 400782

calendar_today

Updated On:

Products

VMware vSphere Kubernetes Service

Issue/Introduction

  • Enable Workload Management stuck in configuring state showing the following error in vCenter

    Resource Type Deployment, Identifier vmware-system-netop/vmware-system-netop-controller-manager is not found.

    Deployed Control Plane VMs
    Master EAM Agent with identifier 2f486a27-####-###-####-#####4b7ac could not deployed. See ESX Agent Manager logs for more details.

  • The vCenter /var/log/vmwar/wcp/wcpsvc.log will show similar events:

    2025-06-05T20:07:25.510Z debug wcp [kubelifecycle/eam_monitor.go:102] [opID=682ea811-ff0d7a19-####-####-####-####0dce0f] Supervisor ff0d7a19-####-####-####-####0dce0f has eam issues [[{1698 *types.Issue {vcenter.wcp.eam.issue.clusterVmNotDeployed Master EAM Agent with identifier 5ad3d107-####-####-####-30093cecf3d9 could not deployed. See ESX Agent Manager logs for more details. [5ad3d107-####-####-####-30093cecf3d9] map[] <nil>} true} {1697 *types.Issue {vcenter.wcp.eam.issue.clusterVmNotDeployed Master EAM Agent with identifier 7ef3bfca-####-####-####-86c7ac0661b7 could not deployed. See ESX Agent Manager logs for more details. [7ef3bfca-####-####-####-86c7ac0661b7] map[] <nil>} true} {1696 *types.Issue {vcenter.wcp.eam.issue.clusterVmNotDeployed Master EAM Agent with identifier 2f486a27-e04d-4b13-a228-9b471354b7ac could not deployed. See ESX Agent Manager logs for more details. [2f486a27-e04d-4b13-a228-9b471354b7ac] map[] <nil>} true}]]
    2025-06-05T20:07:25.511Z debug wcp [kubelifecycle/controller.go:500] [opID=682ea811-ff0d7a19-####-####-####-####0dce0f] Supervisor deployment retry.

  • The vCenter /var/log/vmware/eam/eam.log the following exception:

     2025-06-05T20:17:11.858Z |  WARN | vim-async-2 | EnterVmListener.java | 191 | [PROCESS_VM:52c01559ea9ff4e0] Enter change for missing VM VirtualMachine:vm-2##4
    com.vmware.eam.RemoteObjectMissingException: 8284a868-####-###-####-6d9c83aa8986::VirtualMachine:vm-2##4 Missing object id: null.
        at com.vmware.eam.vim.vm.impl.VirtualMachine.createMissingObjectError(VirtualMachine.java:489) ~[eam-server.jar:?]
        at com.vmware.eam.vim.vm.impl.VirtualMachine.getExcTransform(VirtualMachine.java:394) ~[eam-server.jar:?]
        at com.vmware.eam.async.remote.Completion.onError(Completion.java:86) ~[eam-server.jar:?]
       
    Caused by: com.vmware.vim.binding.vmodl.fault.ManagedObjectNotFound: The object 'vim.VirtualMachine:vm-2##4' has already been deleted or has not been completely created

  • The /var/log/vmwar/wcp/vpxd.log will show recurrent deploying of SupervisorControlPlaneVM events but there will be node SupervisorControlPlaneVM created succesfully. 

    2025-06-05T19:48:53.183Z |  INFO | cluster-agent-4 | DeployVmJob.java | 552 | Deploying agent on cluster WDP_KUBE (ClusterComputeResource:domain-c9) with name SupervisorControlPlaneVM (1132)
    2025-06-05T19:48:53.194Z |  INFO | cluster-agent-1 | DeployVmJob.java | 552 | Deploying agent on cluster WDP_KUBE (ClusterComputeResource:domain-c9) with name SupervisorControlPlaneVM (1133)
    2025-06-05T19:48:53.205Z |  INFO | cluster-agent-3 | DeployVmJob.java | 552 | Deploying agent on cluster WDP_KUBE (ClusterComputeResource:domain-c9) with name SupervisorControlPlaneVM (1134)
     
    2025-06-05T19:49:03.282Z |  INFO | cluster-agent-2 | DeployVmJob.java | 552 | Deploying agent on cluster WDP_KUBE (ClusterComputeResource:domain-c9) with name SupervisorControlPlaneVM (1135)
    2025-06-05T19:49:03.295Z |  INFO | cluster-agent-0 | DeployVmJob.java | 552 | Deploying agent on cluster WDP_KUBE (ClusterComputeResource:domain-c9) with name SupervisorControlPlaneVM (1136)
    2025-06-05T19:49:03.308Z |  INFO | cluster-agent-4 | DeployVmJob.java | 552 | Deploying agent on cluster WDP_KUBE (ClusterComputeResource:domain-c9) with name SupervisorControlPlaneVM (1137)

  • The /var/log/vmwar/vpxd/vpxd.log show the following exception during creating SupervisorControlPlaneVM 

    2025-06-05T19:41:08.143Z info vpxd[05569] [Originator@6876 sub=VmProv opID=6e7077c2-01-01-01] PrevOutputs: (vpx.vmprov.Action.Output) [
    -->    (vpx.vmprov.ReserveDirectory.Output) {
    -->       reservation = (vpx.vmprov.ReserveDirectory.ReservationInfo) [
    -->          (vpx.vmprov.ReserveDirectory.ReservationInfo) {
    -->             parentDir = "ds:///vmfs/volumes/5e457522-####-####-0025b515a004/",
    -->             dirBasename = "SupervisorControlPlaneVM (996)",
    -->             reservedDir = "ds:///vmfs/volumes/5e457522-#####-#####-0025b515a004/SupervisorControlPlaneVM (996)"
    -->          }
    -->       ]
    -->    }
    --> ]
    2025-06-05T19:41:08.135Z info vpxd[05533] [Originator@6876 sub=VAppImport opID=3cf45609-01-01] Exception during CreateVM: N5Vmomi12RuntimeFault9ExceptionE(Fault cause: vmodl.RuntimeFault
    --> )
    --> [context]zKq7AVECAQAAACqAeAEbdnB4ZAAAQxxTbGlidm1hY29yZS5zbwAACBhCACk/QwCWmUoBkxAYbGlidm1vbWkuc28AAXLgI4Lk2SIBdnB4ZACCdvV/AoIybzQCgmJwNAKC0UxHAYKy0IACgv3ggAKC8fN/AoLWUYACgoRLRwGCzW9HAYILdEcBgrLQgAKC/eCAAoLx838CguDKgAIABOw3ABdFOADFD1EDsI4AbGlicHRocmVhZC5zby4wAATf+g9saWJ
    jLnNvLjYA[/context]
    2025-06-05T19:41:08.143Z info vpxd[05533] [Originator@6876 sub=vpxLro opID=3cf45609-01-01] [VpxLRO] -- FINISH lro-2572961
    2025-06-05T19:41:08.143Z error vpxd[05533] [Originator@6876 sub=Default opID=3cf45609-01-01] [VpxLRO] -- ERROR lro-2572961 --  --  -- ResourcePool.ImportVAppLRO: :vmodl.RuntimeFault
    --> Result:
    --> (vmodl.RuntimeFault) {
    -->    faultCause = (vmodl.MethodFault) null,
    -->    faultMessage = (vmodl.LocalizableMessage) [
    -->       (vmodl.LocalizableMessage) {
    -->          key = "com.vmware.vim.vpxd.encryption.noDefaultKmipServer",
    -->          arg = <unset>,
    -->          message = <unset>
    -->       }
    -->    ]
    -->    msg = ""
    --> }
    --> Args:
    -->

 

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

  • VMware vSphere Kubernetes Service (VKS)

Cause

  • The vSphere Supervisor Installation is been performed with using "VM Encryption Policy" storage policy which is currently not supported.

  • The  storage policy  name = VM Encryption Policy with id = 4d5f673c-###-###-###-9e71128cae77 

    +++content-library/cls.log+++

    2025-06-04T17:16:29.745Z | DEBUG    | mawqr82x-56224-auto-17dt-h5:70014352-bc-0c | cls-simple-activity-2     | GetDefaultDatastoreMappingActivity | Default datastore mapping params: DatastoreMappingParams (com.vmware.vcenter.ovf.datastore_mapping_params) => {
        availableStorageProfiles = [StorageProfileTarget (com.vmware.vcenter.ovf.storage_profile_target) => {
        id =4d5f673c-###-###-###-9e71128cae77,
        name = VM Encryption Policy

  • The vCenter /var/log/vmwar/vmware-sps/sps.log show  that SupervisorControlPlaneVM is getting created using storage policy with ID 4d5f673c-###-###-###-9e71128cae77 which is the "VM Encryption Policy" storage policy. 


    2025-06-05T20:03:02.865Z [pool-4-thread-3] INFO  opId=1dde0751-01-01-01 com.vmware.pbm.profile.impl.ProfileManagerImpl - Starting Timer: preProvisionProcess. Entity associations: EntityAssociations{entity = null, entityName = SupervisorControlPlaneVM (1386), operation = CREATE, policyAssociations = [{entity = {key = VMHome}, hub = {hubId = datastore-16}, policySpec = {profileId = 4d5f673c-###-###-###-9e71128cae77}, defaultPolicy = false}, {entity = {key = VMHome:-103}, hub = {hubId = datastore-16}, policySpec = {profileId = 4d5f673c-###-###-###-9e71128cae77}, defaultPolicy = false}]}, UserPrivileges: null

Resolution

Powered by Wolken Software