"Failed to bootstrap VCF services runtime" during VCF/VVF 9.0.x to 9.1 upgrade
search cancel

"Failed to bootstrap VCF services runtime" during VCF/VVF 9.0.x to 9.1 upgrade

book

Article ID: 446929

calendar_today

Updated On:

Products

VCF Operations

Issue/Introduction

  • While upgrading VCF environment from 9.0 to 9.1, the deploy VCF Management components step fails at "Bootstrap VCF service platform" task.
  • The cause section as per the UI would be " java.lang.StringIndexOutOfBoundsException: Range [-1, 0) out of bounds for length 0 Range [-1, 0) out of bounds for length 0"
  • As per the /var/log/vmware/vcf/domainmanager/domainmanager.log
     ERROR [vcf_dm,6a4210ba7a1e89###] [c.v.v.v.services.VspServiceImpl,pool-3-thread-4]  Error: error while validating VC resources: failed to convert datastore: failed to convert MoID Datastore:datastore-## to path: ServerFaultCode: The object 'vim.Datastore:datastore-##' has already been deleted or has not been completely created

     ERROR [vcf_dm,6a4210ba7a1e89###] [c.v.v.v.services.VspServiceImpl,pool-3-thread-4]  error: error while validating VC resources: failed to convert datastore: failed to convert MoID Datastore:datastore-## to path: ServerFaultCode: The object 'vim.Datastore:datastore-##' has already been deleted or has not been completely created

     DEBUG [vcf_dm,6a4210aff5cf8461###] [c.v.e.s.c.util.LocalProcessService,dm-exec-63]  Command exit code: 1

     ERROR [vcf_dm,6a4210aff5cf8461###] [c.v.e.s.c.util.LocalProcessService,dm-exec-63]  Local command  /opt/vmware/vcf/domainmanager/tmp/vmsp1100016129681162### bootstrap cluster deploy --wait /nfs/vmware/vcf/nfs-mount/bundle/aa88f811-###-###-c401919###/######-####-###/vcf-services-platform-template-9.1.0.0.25370367.ova --props-file=/opt/vmware/vcf/domainmanager/tmp/vmsp.props.579564027907227###.tpl failed with exit code 1

    Out Stream:

    Error Stream: LocalProcess ERROR:  CLI Version: 9.1.0.0.25370367

    DEBUG [vcf_dm,6a3a638022a848d###] [c.v.e.s.c.util.LocalProcessService,dm-exec-581]  Error Output:  IP address not specified. Trying to look up the bootstrap VM in vCenter

    Using appliance name from props: bootstrap-vm-Hri###

    Error: failed to find bootstrap VM 'bootstrap-vm-Hri###': failed to find VM /SR##/vm/vcf-management-services/bootstrap-vm-Hri###: vm '/####/vm/vcf-management-services/bootstrap-vm-Hri###' not found

    Error: failed to find bootstrap VM 'bootstrap-vm-Hri###': failed to find VM /SR##/vm/vcf-management-services/bootstrap-vm-Hri###: vm '/####/vm/vcf-management-services/bootstrap-vm-Hri###' not found

    ERROR [vcf_dm,6a3a638022a848dccf4###] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-581]  [DMQQ6A] PUBLIC_VSP_CLUSTER_BOOTSTRAP_FAILED Failed to bootstrap VCF services runtime

    com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to bootstrap VCF services runtime

            at com.vmware.vcf.vimanager.fsm.actions.BootstrapVspClusterAction.execute(BootstrapVspClusterAction.java:218)

            at com.vmware.vcf.vimanager.fsm.actions.BootstrapVspClusterAction.execute(BootstrapVspClusterAction.java:89)

            at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.invoke(FsmActionState.java:66)

            at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:161)

            at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:147)

            at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.invokeMethod(ProcessingTaskSubscriber.java:401)

            at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.processTask(ProcessingTaskSubscriber.java:515)

            at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.accept(ProcessingTaskSubscriber.java:128)

            at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)

            at java.base/java.lang.reflect.Method.invoke(Method.java:580)

            at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:85)

            at com.google.common.eventbus.Subscriber.lambda$dispatchEvent$0(Subscriber.java:71)

            at com.vmware.vcf.common.tracing.TraceRunnable.run(TraceRunnable.java:63)

            at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)

            at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)

            at java.base/java.lang.Thread.run(Thread.java:1583)

    Caused by: java.lang.RuntimeException: java.lang.StringIndexOutOfBoundsException: Range [-1, 0) out of bounds for length 0

Environment

VMware Cloud Foundation 9.x

Cause

VCF components were migrated from one datastore to another which left the SDDC Manager DB with the old datastore MOID. During the deployment, the workflow looks for the datastore MOID in the SDDC Manager DB and the vCenter. Since it is unable to find the datastore MOID in vCenter inventory as compared to what it sees in the SDDC manager DB, the deployment fails.

Resolution

Currently migrating VCF components from one datastore to another from vCenter UI is not supported, as this causes primary datastore MOID mismatch in the SDDC Manager DB. Contact Broadcom support for further assistance. 

Powered by Wolken Software